ΓᾺΡ - eee ὁ, ἢ ἡ. τ ἘΠῚ ΠΥ ΣΥΥΣ ‘ AL) bd ΒΡ. -" ae 9s, Fee -- αν ΓΜ sk . al Se hi hh oe ee ee Rom ἐς ΤΙ Τ᾿ ~ —~ rte. wee, Oe Tl ὉΠ ee ΤΡ ΡΒ a) er oa ΟὟ a τ» 
7 : . Ἶ af ᾿ i Ξ - πῇ 4 
: i eZ | Ρ 2 = ‘ ; = 1 
: i tan hy : a ty Pa 
(y ‘6 / 3 7 , 
ἵ i * 
* - i 
i ‘ 
* ‘ 
i 4 
* 
ts 
" 
= ἢ 
7 j ao 
BS f é 
1 
τι 7 
t 
i a : 
; 3 
" 
᾿ ᾿ a 
ἀ ἃ; Me 
¥ 7 
εἰ Γ᾿ 
i 
5 a 
: : , 
d , : 
: i i ® 
Ἷ “ 
᾿ 
᾿ : 
] 
δ 
; Γ, 
rt ; 
"4 
. 


AEA A ioe i“ ~~, 2 ' “ls 


ERAT Seg = | ~ £3 7 = ee 


᾿ 


NUMERICAL 
METHODS 


By ἐξ 
ANDREW D. BOOTH, D.Sc. 


Dean of the College of Engineering, 
University of Saskatchewan, 
Interdisciplinary Professor of Autonetics, 
Western Reserve University, Cleveland 


THIRD EDITION 


LONDON 
BUTTERWORTHS 
1966 


ENGLAND: 


AUSTRALIA: 


CANADA: 


NEW ZEALAND: 


SOUTH AFRICA: 


U.S. A. ᾿ 


BUTTERWORTH ἃ CO. ἘΠ mealies LTD. 
LONDON: 88 Kingsway, W 


BUTTERWORTH ἢ ζῶ. pe ONT LTD. 
SYDNEY: 20 Loftus Street 
ELBOURNE: 473 once 2 δῦνοι 
BRISBANE: 240 Queen 5 
BUTTERWORTH & CO. [nana LTD. 
TORONTO: 1367 Danforth Avenue, 6 
BUTTERWORTH & CO. (NEW ZEALAND) LTD. 


WELLINGTON: 49/51 Ballance Street 
AUCKLAND: 35 High Street 


BUTTERWORTH & CO. (SOUTH AFRICA) LTD. 


DURBAN: 33/35 Beach Grove 


BUTTERWORTH INC. 
WASHINGTON, D.C.: 20014: 7300 Pearl Street 


First Edition 1955 


Made and printed by offset in Great Britain by 
William Clowes and Sons, Limited, London and Beccles 


CTY OF LIVERPOOL 


Pwvicre A Fr Se. κα Aor & ku 


PREFACE TO THIRD EDITION 


REFLECTING the advances in the subject since the first edition, much 
new material has been added to the text. This new material takes 
account, not only of experience gained in using the earlier edition as 
a class text book, but also of our observations of the virtues of different 
methods on modern computing machines. 

A. Ὁ. Β. 
Saskatoon 


PREFACE TO FIRST EDITION 


THE present book has grown out of the series of lectures given by 
the author to final honours B.Sc.:mathematics students at Birkbeck 
College, London. The purpose of the course has been, not so much 
to instruct in the detailed tedium of actual calculation, but rather 
to give an understanding of the basic principles upon which such 
analyses rest. 

Some existing books on numerical analysis lay much stress upon 
the detailed form in which a given procedure is to be laid out; it has 
been my experience that such concentration upon actual numbers 
obscures the underlying mathematical basis upon which the work 
rests, and so such tabulations are almost entirely absent from this 
book. Where they are given, as in Chapter 7, they illustrate the 
sort of behaviour which will be encountered in a calculation rather 
than any detailed form of layout.": | 

Were these didactic points the sole reason there would be little 
Justification for a new book on computation; a far more important 
consideration lies in the growth, during the past decade, of the 
science and art of programming for an automatic digital calculator. 
The classical methods of hand calculation are, to a greater or less 
extent, unsuitable for the modern machines, and only by having a 
thorough knowledge of the underlying mathematical principles, is 
the programmer likely to make effective use of the new tools. 

At Birkbeck College Computational Laboratory the teaching of 
numerical methods has been accompanied by the actual use of an 
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automatic calculator, and demonstrations of such things as differ- 
encing and the solution of differential equations have been carried 


out by the machine and not by the student. Perhaps not unnatur- CONTENTS 
ally, this has proved more popular than the old method. 

A book of this kind must always owe much to the work of previous PAGE 
authors; it is pleasant to acknowledge the help which the author PREFACE . ᾿ : ; ; : : A : m 
derived from Freeman’s ‘Actuarial Mathematics’ and from the 
classical ‘Calculus of Observations’ of Whittaker and Robinson, |. THe NATURE AND ΡΌΚΡΟΒΕ OF NUMERICAL ANALYSIS. 1 
both of which were practically the only available works during the 
1930’s. In more recent times the paper on ‘Difference and Associ- 2. TABULATIONS AND DIFFERENCES . . . . . 7 
ated Operators’ by W. G. Bickley may be mentioned as having 
particular influence. 3. INTERPOLATION . Ξ ; ὦ . ; eee 

Finally it gives me particular pleasure to acknowledge the help of | | 
my wife both in making the book more readable than might other- 4. ΝΌΜΕΒΙΟΑΙ, DIFFERENTIATION AND INTEGRATION . . 30 
wise have been the case, and also, in collaboration with J. P. Cleave, | 
B.Sc., for checking the examples given in Chapter 7. 5. THE SUMMATION OF SERIES. . - . . . 54 
ἀρ: ad 6. Tse SoLution or Orprnary DirFERENTIAL Equations. 59 
Fenny Compton 

7. SmmutTANeous Linear EQuaTIons ‘ ,| oa 
8. PARTIAL DIFFERENTIAL EQUATIONS. : . ν 124 
9. Non-LINEAR ALGEBRAIC EQUATIONS. 3 3 . 156 
AUTHOR’S NOTE 10. APPROXIMATING FUNCTIONS. ; : . : . 178 

Advantage has been taken of the need for a second edition to i). ‘Sees Soseruen dies Annes 187 
correct small errors which were present in the earlier version, and 
the author wishes to express his thanks to reviewers and others who 12. InrecRAL EQuations. . 197 
drew his attention to these. | 

A. 8.8. νον Βυυδσδνην lw ll lt lel tlt 55 

Fenny Compton 
Name INDEX . ; ‘ ‘ : ‘ ‘ , , 209 
ΘΌΒΙΕΟΤ INDEX . : ‘ i ‘ ‘ ᾿ : . 211 
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I 
THE NATURE AND PURPOSE OF 
NUMERICAL ANALYSIS 


1.1 HISTORY 


ALTHOUGH numerical analysis is considered by some to be a subject 
of recent origin and development this is not, in fact, so. Dealing, as 
it does, with the derivation of results in the form of numbers, the 
numerical analyst is really the lineal descendant of the first caveman 
who enumerated the number of his wives by putting them into 
one : one correspondence with the fingers of his hand. 

Even in its more modern aspects the subject is antique; thus a 
primary activity of the scientists of Babylon was the construction of 
mathematical tables. An example is extant, dating from about 
2000 3.c., which contains on a tablet the squares of the numbers 
1-60. Another tablet records the eclipses going back to 747 B.c., so 
that astronomical calculation formed a part of the activity of 
these early numerical analysts. 

The ancient Egyptians, too, were energetic numerical analysts. 
They constructed tables whereby complex fractions could be decom- 
posed into the sum of simpler forms with unit numerators, and 
invented the method of false position (see Chapter 9, section 9.3) for 
the solution of non-linear algebraic equations. 

Passing to the Greek mathematicians we find Archimedes, in about 
220 B.c., approximating the value of 7 and describing it as less than 
3} but greater than 34. Heron the elder, in about 100 .c., made use 


of the iterative process: 4/a~ a(x = ) which is usually ascribed to 
nm 


Newton, and the Pythagorean school considered the summation of 
the series (1 + 2+ 3...). Diophantus, about a.p. 250, apart from 
his better known work on indeterminate equations, was responsible 
for a process for the arithmetical solution of quadratic equations. 
The Hindus were the creators of our modern arithmetic notation 
—usually called Arabic—and devised the method of checking the 
correctness of an arithmetic calculation known as ‘casting out nines’. 
Mohammed ibn Musa Al-Khowarizmi was the first Arab arith- 
metician and was responsible, around a.p. 820, for the systematiza- 
tion of computational processes. He gave the value πὶ = 62832/20000 
and was active in the preparation of astronomical tables. Abul Wefa 


1 


NATURE AND PURPOSE OF NUMERICAL ANALYSIS 


(A.D. 960) devised a method for the computation of tables of sines 
and gave the value of sin (4°) correct to nine decimal places; he also 
used the tangent and calculated a table of this function. 

Jumping to the seventeenth century, it is interesting to note that 
Napier’s first table of logarithms was produced before the use of 
exponents was current, and that his ‘logarithm’ differs from any in 
current use since: 

Naperian log x = 10? log, (107/x). 

In 1614 Napier published his Mirifict logarithmorum canonis de- 
scriptio and, posthumously, his Mirifict logarithmorum canonis constructio 
in 1619. Briggs, only slightly later in 1624, produced his Arithmetica 
logarithmica, which contains the logarithms, to 14 places of the num- 
bers 1-20,000 and 90,000—100,000. Vlacq produced, in 1628, a table 
which is still fundamental of the 14-place logarithms of the numbers 
1-100,000. The first authoritative publication of the logarithms of 
trigonometric functions was made at about the same time (1620), by 
Gunter, who invented the words ‘cosine’ and ‘cotangent’, and was 
responsible for the so-called ‘Gunter’s chain’. 


In the nineteenth century there occurred one of the triumphs οὖ 


numerical analysis, the simultaneous prediction by. Adams and le 
Verrier in 1845, of the existence and position of the planet Neptune. 
This century saw also the rise and development of automatic calcu- 
lating machinery, from the crude desk multiplier of Thomas de 
Colmar to the almost unmodified Brunsviga of the present day, the 
Hollerith punched card census calculator, and the difference and 
analytical engines of Charles Babbage. 

Not until the end of the 1930's did the fully automatic calculators 
begin to come into use, and since the late 1940’s there has been a 
revolution and renaissance in numerical analysis. New methods 
have been developed and problems which could not previously have 
been contemplated, even for a life’s work, are now solved in hours. 
It is perhaps dangerous to quote examples, but outstanding achieve- 
ments are the calculation of πὶ and ὁ to 100,000 decimals’ which took 
an I.B.M. 7090 computer just under 8} ἢ. The demonstration of the 
primeness of the Mersenne number 2?*7° — | on $.W.A.C. in 13 min. 
25 sec., may also be cited as a noteworthy achievement. 


12 PHE TOOLS OF ANALYSIS (HAND) 
From the classical standpoint of the individual numerical analyst 
the tools of computation are: 


(1) Tables of formulae (3) A desk calculator 
(2) Tables of function values (4) Pencil, paper and rubber 


TOOLS OF ANALYSIS (HAND) 


Few investigations are of such a fundamental nature that they make 

no use of existing mathematical knowledge; probably the most com- 

mon table of formulae in use is a list of integrals. Four standard 

works may be mentioned: 

(1) Perrce, B.O., ‘A short table of integrals,’ Ginn, Boston (1929) 

(2) Dwicnt, H. B., ‘Tables of integrals and other mathematical 
data,’ Macmillan, New York (1934) 

(3) pe Haan, D. B., ‘Nouvelles tables d’intégrales définies, re- 
printed Stechert, New York (1939) 

(4) * Interpolation and allied tables,’ H.M. Stationery Office (1956) 

The first two volumes are chiefly concerned with indefinite integrals, 

and the third exclusively with definite integrals. The last booklet 

contains most of the useful formulae for interpolation. 

Tables of function values are almost too numerous to mention. For 
4- or 5-figure accuracy there are the classical: 

JAHNKE-Empe, ‘'Tafeln héherer Funktionen,’ Teubner (4th edn.), 
Leipzig (1948) 

Empe, ‘Tafeln elementarer Funktionen,’ Teubner, Leipzig (1940) 
which, besides giving numerical values, contain useful graphs and 
formulae, Other moderate accuracy collections are: 

‘Mathematical Tables from the Handbook of Chemistry and 
Physics,” Chemical Rubber Publishing Co. Gleveland (1946) 

Date, J. B., ‘Five-figure Tables of Mathematical Functions,’ 
Arnold, London (1937) 

Dwicut, H. B., ‘ Mathematical Tables,’ McGraw-Hill, New York 
(1941) 

More accurate tables (6 or more decimal digits) are: 

“Chambers’s Seven-figure Mathematical Tables,’ Chambers, 
London (1937) 

“Chambers’s Six-figure Mathematical Tables’ (2 vols.) (Ed. 
Comrie) London (1948-9) 

“Barlow’s Tables of Squares, Cubes and Reciprocals’ (Ed. Comrie) 
Spon (1941) 

For more specialist tables reference can be made to the monu- 
mental: 

FLETCHER, A., Mixxer, J. C. P., RosENHEAD, L. and Comrie, L. J.; 
‘Index of Mathematical Tables,’ 2nd edn., Addison-Wesley, Mass. 
(1962) 

It may be supplemented by reference to ‘Mathematical Tables and 
other Aids to Computation’ (.!f.7.4.C.) which maintains a cumu- 
lative description of new tables. 


NATURE AND PURPOSE OF NUMERICAL ANALYSIS 


Numerous desk calculators are now available but none can be 
said to possess such excellence as to satisfy all felt wants and to out- 
shine the others. Our experience recommends the Brunsviga and 
Madas machines in the hand-operated range, and the Marchant, 
Madas and Mercedes-Euclid amongst those electrically operated. 
We shall not attempt any description of the means of using these 
machines, since a short time with an experienced operator and the 
machine will do more than many pages of words in this direction. 

Regarding pencil and paper, we may remark that foolscap paper 
ruled with feint } in. squares seems convenient for most general 
purposes. Pencils should be soft, B is suitable, and the rubber should 
not be one degraded by age and use ! 


13 THE TOOLS OF ANALYSIS (AUTOMATIC) 


It is hoped that amongst the readers of this book there will be many 
who have access to one of the automatic digital calculators which 
become daily more easily available. 

Fortunately most of the available automatic machines are provided 
with an autocoding system such as Fortran or Algol. ‘The develop- 
ment of these autocodes has revolutionized the preparation of 
arithmetic problems for computation, The compiling programmes 
can, to a large extent, detect careless errors in preparation and the 
very extensive facilities for floating point arithmetic, which are built 
into modern autocodes, make the careful experimental arithmetic of 
the immediate past almost unnecessary. 


1.4 PRECISION, ACCURACY AND ERRORS 


In planning a calculation the three factors detailed in the heading 
must always be considered. First, in any calculation using data ob- 
tained by physical measurement, the inherent precision of the data 
themselves must be examined. If no figures for experimental errors 
are presented and these are not easily obtainable, a knowledge of the 
experimental technique may give a clue. Measurements of length 
are rarely accurate to better than 1/10 per cent, measurements of 
weight often attain 1/10,000 per cent. Electrical measurements are 
frequently of precision as low as 5 per cent. These circumstances 
should be taken into account at the planning stage, and a rough 
working rule is to calculate to two places more than those given by 
the data. 

The accuracy of the calculation (excluding errors of the careless 
type) will depend on the numerical process involved. Additions and 
subtractions neither increase nor decrease the precision of the data; 
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multiplications and divisions, however, lead to round off procedures 
and thus to an overall decrease in accuracy. Hand calculations are 
seldom of such length as to cause trouble from the growth of round 
off errors, but with automatic calculators the situation is different. 
Thus a typical matrix inversion may lead to over 10,000 multiplica- 
tions, and this, in turn, to a rounding error which has a probable 
value of the order of 100 units in the last place. 

Errors are of two main types, mathematical and human. The 
former may result from the use of approximations, which are inevit- 
able consequences of the use of discrete processes to represent con- 
tinuous ones. The latter class of error should be avoidable, at least 
in the long run, by the provision of adequate checks. 

In manual calculation it has long been a platitude that a person 


should never check his own work and that, if possible, the same 


method should not be used. This has tended to become forgotten in 
connection with the use of automatic digital calculators, and it is 
frequently suggested that because the machine has produced the 
same answer twice in succession it must be correct. Unfortunately 
most machines suffer at times from ‘pattern sensitivity’, that is they 
will work with complete accuracy on all numbers except one. Under 
these circumstances the same wrong answer can be produced as often 
as required, and the only valid check is a completely different com- 
puting routine. 

In practice it is often possible to check the results of a long series of 
calculations by some completely external means, such as differencing 
(see Chapter 3, section 3.2), or, alternatively, from a knowledge of 
the observations with which the calculations are intended to agree. 
When this is not so (with an automatic digital calculator) a good plan 
is to repeat the calculation after an interval of several days, since 
few, ifany, of these machines survive such a period without servicing 
and this almost always varies any pattern sensitivity which may be 
present. 

Formulae may be used in different ways and with differing re- 
sultant accuracy. Thus, we may calculate 


sin θ = 0 — @/6 + 65/120 soe (TAD 


in two ways. In the first the terms are formed separately and then 
added. In this event the round off may be equal to 1-5 in the last 
place. On the other hand, by forming: 


[{τῈ|593 — 2603 + 110 ....{({1.4.2} 
the greatest error will be 0-5 in the last place. This example is also 
instructive in that it illustrates an efficient means of calculating a 
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polynomial. Thus a direct calculation of equation 1.4.1 involves two 
divisions, four multiplications and two additions or subtractions, 
whereas in equation 1.4.2 the multiplications are reduced to three. 
Considerations such as those mentioned above should always 
precede any numerical calculation and might be multiplied indefi- 
nitely. Some pointers will be given at appropriate places in the text, 
but pencil and paper analysis can only be learned by long practice 
and, in our experience, no two expert analysts agree upon the best 
detailed layout for any particular case. For this reason the details 
will be left to the reader and such numerical examples as appear 
are illustrations of such things as convergence, rather than of com- 


putational layout. 


REFERENCE 
™ Snanxs, Ὁ. and Wrencu, J. W., Maths Comput., 16 (1962) 76 


2 
TABULATIONS AND DIFFERENCES 


2.1 THE NATURE OF TABULATED FUNCTIONS 
Tue differential calculus had its origin in a consideration of the 
mode of variation of a function γ = f(x) with the augument x. In the 
process of defining the differential coefficient, <4 it is necessary to 
consider the limit of a finite difference ratio: 
f(x + 8x) — f(x) /(* + 8x) — x 

as ὃχ -» 0. When a function is represented by a set of numerical 
values contained in a table, it is natural to consider the analogues of 


the differentials dy and dx which can be deduced immediately from 
the tabular values. Suppose that a function u, is defined by a table: 


x ly 
0 ly 
] uy 
2 | ly 
nm | tp 


Then, corresponding to dx, we have (1 — 0), (2 — 1), (n — {n — 1}) 
all of which are equal to unity. And corresponding to dy we have the 
differences (wu, — up), (ug — 1%), . » + (Up — tp). 

Since the interval of tabulation (i.e. 1 in this case) is by no means | 
always unity, it is customary to represent this by the symbol 4 (x), 
and in a like manner: πέρι. — Up is represented by 4 (tp). 

Now just as it is possible to proceed to differential coefficients 
higher than the first, so can the differences of a tabulated function 
be extended thus: 
πὴ ΔΙ Δ" | “ihe | 
Ow, Ju=u,—u, A%,=—Au, —du,— w,— Iu, +H, A, =A", —A«, = uy — Bu, + 3u, —u, 
1), “νι; τὰ μα τον 4%4,—Au,—Au,=4,—2u,—u, 4%,=4%,—A%, =u, — Sly + Sty — Ν᾿ 
2. ὦν duou,— uy, ete. etc. 

Siu, du,=u,— μα 
4ju, du, =u, —w, 
blu, duy= μὲ -- uy, 

The differences on the first line, namely, 4u,, 42u,, Au, etc. are 
referred to as leading differences. 
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2.2 SOME ACTUAL TABLES 


Before proceeding to a consideration of the uses to which differences 
may be put, it is instructive to consider some existing mathematical 
tables and their relation to differences. 

As a first example consider the following section of a typical 
schoolboy’s 4-place logarithm table: 


x o 1 8 &8 ploy yas Se eee eee 


10 | 0000 0043 0086 0128 0170 | 0212 0263 294 0334 0374 | 4 8 123 17 21 25 29 33 37 
11 0414 0453 0492 0531 0569 | O607 0645 0682 0118 O755 | 4 8 1115 19 25 26 30 34 
12 0792 0828 0864 0899 0934 etc. ele. 


This is a typical example of what may be called a ‘two dimensional’ 
table in that, to each lattice point of a two dimensional co-ordinate 
system is assigned a functional value. Such tables are usually only 
available for the most elementary functions and at a very low level 
of precision. First consider the differences of the function at the 
finest interval available in the table, namely -001. 


x F(x) A A? 
100 0000 0043 0 
101 0043 0043 — | 
102 0086 0042 0 
103 0128 0042 
104 0170 


It is intuitively clear that, since the first difference is sensibly 
constant, a linear relationship exists between the function and its 
argument in the intervals between tabulated values. It follows that, 
at any rate approximately, if the value of f(x) at a non-tabulated 
value is required no great error will result from assuming: 


7. +8) =f(%o) + δι /(%) —S(%0)] 
λυ + 8) =f(%o) + 8-ALf(%0)] 


where ὃ is a proportion of the interval (x, — xy), and in any case the 
error will not exceed unity [i.e. the value of 4*{ f(x,)}]. On the other 
hand, consider the table of ‘proportional parts’. These purport to 
contain the values of 34[( f(x 9)] etc. for ὃ = -0001, -0002 . . . -0009. 
But since: 


or 


Af(100) = 0043 
Af(109) = 0040 
it is evident that the proportional parts can only be approximate and 
8 


SOME ACTUAL TABLES 


that the value at ὃ = -0009 should be 39 for x = 1009 and 36 for 
x = 1099. More honest table makers would mark these proportional 
parts to indicate that they are not accurate over the whole range of 
argument which they cover. 

Next consider the differences over a larger interval (01). 


x | "Ὁ Δ 4 Δ: 
10 | 0000 0414 — 0036 + 0005 
1 | 0414 0378 — 0031 + 0006 
12 | 0792 0347 — 0025 + 0003 
13 | 1139 0322 — 0022 
14 | 1461 0300 
15 | 1761 


It is seen that the first differences are no longer constant so that an 
attempt to evaluate /(103), say, by the ‘proportional part’ technique 
would not be justified (it would lead to -3 x 0414 = 0124 which 
does not agree at all well with the actual value 0128), and a more 
sophisticated technique would be needed to make full use of the 
4-place tabular accuracy. 

The virtue of the simple table, described above, lies in the ease 
with which it may be used; when more accurate values are required, 
however, the two dimensional layout is no longer possible. Thus, to 
continue further the logarithm table to an accuracy of 7 decimal 
places, and to have direct reading of a 7-place argument would 
require a volume of some 2000 pages! 

To overcome this difficulty, the onus of calculation of intermediate 
values is placed upon the user, and the interval of tabulation is so 
chosen as to make possible the linear interpolation process discussed 
above. (Linear interpolation is the technical term for the ‘pro- 
portional part’ technique just examined). An example from a 7- 
decimal place logarithm tabulation is the following: 


x log x | Diff. 
268 11 428 3130 162 
12 3292 162 
13 3454 162 
14 | 3616 162 
15 3778 162 
16 3940 162 
17 4102 162 
18 4264 162 
19 4426 162 
20 4588 162 
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At a later position in the compendium from which this example is 
taken is to be found a table of proportional differences corresponding 
to all differences encountered in the main table, which in this case 
range from 434 to 43. A section of this table is: 


Diff. | Prop. 1 ae oe’ Gee ere 7 8 9 
162 16 32 4. 65. δ᾽ 97 13 18 148 
168 Ι5 38 4. 65. 82. 98 4 130 147 
164 | 16 33 ὁὃδὁ}«λ λοϑδ 5 ἃ 138 3 Μμ8 
165 | 7 33 50 6 8 98 16 182 149 


Thus, to find log 2-68143, the user of the table has to form the sum: 


log 2.68 143 = log 2-6814 + -3 x -000 0162 
= +428 3616 + -000 0049 
= "428 3665 


If greater precision is required, and careful consideration is needed 
to justify it in any particular case, the function difference must be 
actually multiplied by the proportional part. Thus: 


log 2-681 432 = -428 3616 + -32 x -000 0162 


It should be noticed that when the ultimate accuracy is required, as 
in this case, there is a possibility of an error of + | in the last place 
due to round off in the original table construction. This effect is 
eliminated in the so-called ‘critical tables’ in which the range of x 
for which f(x) has a given value is specified. These tables are, how- 
ever, comparatively rare and are unlikely to come the way of the 
student. 

Our final example is taken from a recent™) table of high precision 
(15:decimal places) for the trigonometrical function sin x° 


χϑ sin x δ᾽ 

17.60 0.30236 98907 50445 #§§ --- 92 10714 
.6] .30253 62492 99766 92 15781 
+62 .30270 25986 33306 92 20848 
+63 - 30286 89387 45998 92 25916 
-64 + 30303 52696 32774 92 30982 

17-65 0-30320 15912 88568 — 92 36048 
“66 ete. 


It will be noticed that, to this precision, no attempt could be made 
to subdivide at an interval sufficiently small to make possible linear 
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interpolation. Instead second central differences (see section 3.4 
infra) have been given and these make possible a reasonable inter- 
polation process (that of Everett, section 3.4, equation 3.4.8) for the 
evaluation of intermediate values. 


REFERENCE 


(1) Table of Sines and Cosines to 15 decimal places at hundredths of a coneee> 
U.S. Nat. Bur. Stand. Applied Mathematics Series. No. 5, Washington (1949) 
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3 
INTERPOLATION 


3.1 NOTATION 
Tue reader will be familiar with the notion of a difference as defined 
in section 2.1; this is not, however, the only type of difference which 
is convenient in numerical work. The notation: 

Aun = Unity — Un -++ (3.1.1) 
is usually referred to as the ‘forward’ difference at u, for a reason 
which will be apparent from the following scheme: 


| 38) 4f(x) Δ) Δ) 

0 | Uy 

A ee 
| iy Wy 

| i ee τοι — Aju, 
2 Uy 4%, Ως 

| Au, Aju, 
3 Us ὲ 4 "ity 

du; 

4 i ly 


which represents, a convenient method of tabulation. 

On the other hand, it is equally possible to form the differences 
(ug — uy), (μι...) — Ug), (Uu-g — u_3) elc., and it is convenient to 
have a ‘notation’ for these, although it must be borne in mind that 
they do not really differ from appropriate entries in the forward 
difference table. The customary notation for the ‘backward’ 
difference is 


Vuln = Un — Un-1 ὑφ 50 8..8} 
and its position in tabulation is shown below: 
- | J) We VIO Vx) 
—4 U_, 
Vu_s 
-- 3 u_s V*u_, 
Vu, V*u_y 
== | U_» Vin, 
Vu_y V 4uy 
-.} δ... V ἴΩ = 
io 
0 | tg “- 
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At this point it should be noticed that the forward difference is 
particularly appropriate at the start of a table, and the backward 
difference at the end where, in the absence of analytical knowledge 
about the nature of the tabulated function, forward differences are 
not defined. 

For intermediate points a third type of difference is appropriate, 
this is the so-called ‘central difference’ defined by : 


Sun = Unty — Un-| .++(3.1.3) 
The table then becomes: 


Se νῦν ee | Se 
—2 Uy» ἢ ' 
ὅμ. ἢ 
— |] Uy δῖω ) 
du_ 4 ὅϑω..} 
0 oe, Uo a ΟΝ... ἊΨ ἐν 1 “uy 
| δι | δϑμλ vo) 
] uy 5 5%u, 
2 | Us ᾧ 


which suggests the appropriateness of the central difference in the 
body of a table. 
A further operator, which is of great utility, is defined by: 


Eun = Un yy +++ (3.1.4) 
Ef(x) = f(x + 8x) 
Successive application of 3.1.4 leads to the symbolic equation: 


EM in = Uni-m ....(3.1.5) 


Again, 3.1.1 can be re-written: 
Un +1 = Uy + Au, 


Eu, = (1 + A)u, 


E=1+4 «ον 5.8.8} 


This relation is one of great power in deriving relationships between 
a function and its differences. 
In a similar manner, from equation 3.1.2: 


Νὰ, = Uy — πῃ 
V=(E-D/E οὐνἴ8.1.2) 
13 


or 


whence, formally, 


or 


or 
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And, from equation 3.1.3: 
du, = Etu, — E-tu, 
or 
ὃ = Εἰ --Ετὶ ...».{..1.8} 


A further operator which is in common use is the ‘averaging 
operator’ μ, this is defined by: 


[tly = ¥ (Un 44 + Un-4) 
w = ζ(δ + Ε-ῦ ... 3.1.10) 


The use of the operator Εἰ is not the only method of forming 
S(* + 8x) from f(x). Thus Taylor’s theorem gives: 
| | Sx)? 
7.α + 8x) = f(x) + δα (Δ) + ἕν (ἡ Ἑ ... 


or, writing the differential in operator form such that: | 


ΜΡ Ἢ Ὁ} 
whence: 


Γ΄) = Df(e) 
f(x) = DF (*) 
(8x)? 


T(x + 8x) = f(x) + dxDf(x) + ΚᾺ D*f(x) ++... 
whence, symbolically: | 


ΕΠ -ἰ} + 8D + SS a . | fe) | 
or 
E = =P τς {3.3.1} 
The operator (6x)D is sometimes represented by 
U = (x)D ....(3.1.12) 


3.2 sOME EXPANSIONS 


We shall now use the notation and operators defined in section 3.1 
to obtain a number of useful results. In a short account, such as the 
present, it is not possible to justify the operational procedures 
rigorously, but the reader can, if he so desires, refer to one of the 
standard texts, mentioned in the references, for a complete demon- 
stration. In the meanwhile, it is worth mentioning that the opera- 
tional method enables most of the standard finite-difference formulae 
to be worked out quickly and, as such, is a great atde memoire when 
no compilation is available. 
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SOME EXPANSIONS 


First let us attempt to evaluate 4"u, in terms of the functional 
values, Up, u, . . . etc. We have: 


A” (ug) = (Ε — 1)" 


[e-nesetale | 
a sa Aa! πὶ, (—1)* 


(n — 1) 
= 


= Uy — ΤΊ Κη τ $e Un— ge « (— πω ,.εο3...}} 


A standard notation for the rth binomial coefficient is given by: 
n(n — —2)...(a—r+ 
eye ee eS eer 
so that equation 3.2.] takes the simple form: 

Aug = Uy — Mun. + Mgln—g.- . (— 1)'Mipttn—y - » - (— 1) "utp 
..«.«.(3.3.8} 
At this point it is worth pausing to discuss the application of 
differencing techniques to the detection of errors in tabulated 
functions. It is clear from equation 3.2.3 that if the mth tabulated 
value of a function is in error by a small quantity ε, the effect on the 
differences will increase in a regular fashion and, for the nth differ- 
ence, will have a maximum effect along a central difference line 
pointing to the incorrect value. This is shown in the section of table 
below, where the presence of an error in the functional value 
at x = 88:55 is clearly indicated. The magnitude and mode of 


"8 
6 
"7 
8 
9 


"6280 


INTERPOLATION 


propagation of the disturbance produced by a single unit error is 
shown more clearly below: 


ri ὃ 6? 53 δι 
0 
0 
0 
0 0 
0 l 
0 1 
0 1 —4 
l —3 
l —2 6 
0 1 ; —4 
0 0 1 
0 
0 0 
0 


A second expansion is obtainable as follows. Consider u,, this 
may be written: 


Un = ἔπι = (1 + 4) "up 

= tly Ἔ mAlg + Aug 2. . + nAtug...+ Ag  «.... (3.2.4) 
from which it can be seen that if the first n leading differences of a 
function are given (for a known interval), and if the initial value is 
known, n tabular values are calculable. An extension of this occurs 
when the nth differences of a function are known to be constant. In 
this event all differences of order greater than n are zero, and 3.2.4 
thus enables a complete tabulation to be made. Actually direct ap- 
plication of 3.2.4 is seldom made since it is easier to proceed directly 
from the table; this appears below in the tabulation of x*, a function 
for which third differences are constant. 


x x? A 4? 4 A‘ 


INTERPOLATION FORMULAE 


The assumption of constancy of rth differences is easily seen to 
imply that u is a function of degree r in the interval. This is 
clear from equation 3.2.4 since, when Aug = const, “ΓΈ ἕω = 0 
(kK=1... ), and thus n,,, will be multiplied by a zero co- 
efficient. Since n, is of degree r in the interval, the result follows. 
An alternative demonstration is the following: 


Ax = [x + (8x) 1" — x 
= r(5x)a?—! + O(x"-3) ....(3.3.5} 
Thus the operation 4 on x’ lowers the degree of the function by unity. 


Repeating r times: 
Ax! =r! (δι) ee 
which is the required result. 
A frequently used notation in finite difference work is 
x) — x(x — 1)\(x — 2)... (x —m+1).... (3.2.7) 
we leave it as an exercise to the reader to prove that: 
AxX™ = τυ eee + (3.2.8) 


3.3INTERPOLATION FORMULAE 


We have seen that the values of a function at the points which are 
integral multiples of the difference interval are obtainable from 
equation 3.2.4. The assumption is that the function is representable 
by a polynomial of degree r in the interval (rth differences assumed 
constant), and, if it can be taken that between tabulation intervals 
the same polynomial is an adequate representation of the function: 


ὡς = ty + Ey + EA%y +... «0+. (3.31) 


de x/3x(x/8x —1) .. . (x/8x —r + 1) 
nem r! 


and 6x is the interval of tabulation, so that &, = x, when 6x = 1. 


where 
.««.{98.3..2} 


It must be made quite clear, however, that this result, known as 
the ‘Newton-Gregory’ interpolation formula, does make the assump- 
tion of polynomial representability. To emphasize this point more 
strongly, assume that r +1 tabular values are given, so that no 
differences above the rth are calculable. It follows from equation 
3.2.4 that 


r 
u, = Ln A*tly 
s=0 
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Now we may add to μη any function: 


ᾧ = n(n — 1)(n — 2) se (n — σ)ψ(π) 


where y(n) is arbitrary, since ¢ is zero at all tabulated values and 
will not, in consequence, be observed. Unfortunately, however, ¢ 
will not, in general, be zero at non-tabulated values, e.g. at u, 
(0 <x <1) and any use of the Newton-Gregory formula in these 
circumstances will lead to error. 

Even when u, is given at all points (— οὐ <n < 00) for integral n 
there is still the possibility of an added function: 


oO = 
φ = La sin πίη 
t=0 
so that care is required here also. 

The reader may consider that these cases are exceptional, but the 
following example is one which, in modified form, might be en- 
countered in practice, and for which Newton-Gregory interpolation 
is impossible. 


x | flx) 4 A? 43 4: 
0 ] 3 9 27 81] 
1 4 12 36 [08 

2 16 48 144 

3 64 192 

4 256 


We require to find f(}), and the Newton-Gregory formula gives: 
fa) 149.54 ΘΟ ἢ 94 DOMED oy 
εὐ δ ὃ gy, 


l+i-t+# — 3... 


i 


from which it is evident that the series diverges and will never pro- 
duce the correct value (4)* = 2. 

The Newton-Gregory formula will, in general, fail when the 
function cannot be expanded in a Taylor series, but in any case it 
is rarely useful in practice because it makes use of tabular values in 
advance of the point at which a function value is required. This is 
shown in the scheme: 
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— | May Au.» | Atu_, 


ον # Gauss ‘backward’ 


2 μΞ 
Au ΞΔ πο "ὡς Gauss ‘forward’ 


2 

3 Uy Au, Newton-Gregory 
Aus 

4 uy 


A more reasonable and convergent procedure would be to make 
use of functional values which straddle the point at which interpo- 
lation is required. Two such schemes are shown in the diagram, and 
these lead to the Gaussian interpolation formulae: 

Uz = Uy + χάρο + Xg4*u_, + (x + 1)34%u_y + (+ 1) δε. +... 
saiwia4 eee) 


which is the Gauss ‘forward’ formula, and: 


p= Ug-+- x4u_,+-(x-+1),4 2u_,+(*+1),43u_,+ (x +2) Atu_.+ ... 

....(3.3.4) 
which is the Gauss ‘backward’ formula. x, has been defined in 
equation 3.3.2. The formulae are also correct when x is the propor- 
tional part of the tabulation interval. 

It has been remarked by Comrie that these formulae, and also 
that of Stirling are never used in good modern practice, and for this 
reason we do not give a detailed proof. The reader will satisfy 
himself as to the correctness of the first few terms in the ‘forward’ 
formula by substituting from: 

Au, = A*u_, + A*u_, 
A*uy = A®u_, + Δι.) 
Atu_, = A‘u_, + A’u_, etc. 
in the Newton-Gregory result (3.3.1) and rearranging. The ‘back- 
ward’ formula is likewise obtained by substituting from : 
Au — Au_, 4 A*u - 
A*uy —_ A*u_, + A*u_, 
Au_, = A%y_. + Atu_, etc. 
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A general derivation has been given by FREEMAN(®), 
If we take the mean of the two Gauss formulae (3.3.3 and 3.3.4) 
we obtain: 


ly = 
3 yee 19) oe 
up + $x(Auy + Δι...) ἘΚ] =A Uy + - SACs aa (4%u_, + 45u_,) 


a a, πλοῦ a ac +e. 


+... PPR ee) 
which is known as Stirling’s formula. 

We now proceed to derive the two formulae which are of most 
common use in actual problems. Since they appear at this point 
they will be expressed in terms of the forward difference operator 4. 
It is, however, more usual to work in central differences and an 
alternative proof in terms of these operators will be given later, in 
section 3.4. 

First we transform the origin of the Gauss backward formula from 
Us to u, so that x becomes (x — 1) and: 


Uz = ἴῃ + (x — 1)Muy + - 1d le τ At + So ae 


τ π΄ “N= 9 Atnat... το 6.5.8) 
Taking the mean of equations 3.3.6 and 3.3.3, 
Ι 
le = H(t, + te) + τ By + ESO (tug + tag) + 


,. .(5.3.7) 
which is Bessel’s formula. 
Finally,“ we may write the Gauss forward formula (equation 
3.3.3}: 


Wy 42 Ξε W, +x, + χμάβωρ + (x + 1)544w + (x + Ἰ),Δ 1... +... 
and similarly equation 3.3.6: 

Wz = W, + (x — L)Awy + %24 2wy + x44 3w_, + (x + 1) 4 4w_, 

Now if we subtract εὖ, from w,,, we obtain: 

Aw, = xdw, + (x + 1),4%w) + 
— (x — 1l)Aw, — x,45w_, — 
(since w,4, — w, = Aw, by definition) 
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whence, on placing εἰς = 4w,: 
αν = x, + (x + IgA%y +... 
— (x — 1)ug—x,42%u_,... 


or 
κ, πα + tg to τῷ τ Thy 


e e- De ηντι Atu_y... ..+.(3.3.8) 


which is Everett’s formula. 


3.4 CENTRAL DIFFERENCES - 


It is clear fom the analysis of section 3.3 that a difference interpola- 
tion formula, for use in the body of a table, suggests the notion of 
central differences (which were mentioned briefly in section 3.1), 
We shall now examine an operational method for obtaining central 
difference formulae in a natural manner. 

First observe, from the central difference notation table of section 
3.1, that the only central differences of uy, u, which are available 
from the tabular values are the even ones 82, 54 etc. Let us, therefore, 
attempt to find an expansion for uw, in terms of 57"u, and §7"u,. 

Operationally we have, 


tly = Εἴ = e*" .u, > wise (oad) 


where U is defined by equation 5.1.12. 
Assuming that an expansion is possible in terms of 8?"u, and §7"u,, 
this will be of the form: 


F(8) .uy + G(8)u, ...«.(3.4.2} 
where F and G are even functions of δ. 
Now 
a: Fu, = οὕ, vee» (3.4.3) 
whence, from equations 3.4.1, 3.4.2 and 3.4.3: 
eU = F(S) + Ο(δ)εῦ 20+ (3.4.4) 


Now from equations 3.1.8 and 3.1.11 we have: 


ae Β U | 
5 = sinh (5) are ὁ. Ἐς ὅτ 
so that 8 is an odd function of U. Now, since F and G are even 
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functions of 5, they are also even functions of U. Thus, replacing 
U by its negative in equation 3.4.4, we obtain: 
e-7U — F(8) + G(d)e-7 +++ (3.4.6) 
and, after some manipulation: 


F(8) = sinh (1 — x)U/sinh 4 


G(6) = sinh xU/sinh U -+++(3.4.7) 


From these expressions, together with equation 3.4.5 it can be 
shown that: 


F(8) = (1 πὶ + Π 155 ἘΝ 82 


PH Sate eens | 


and that 
α΄ 1ἡ 8... 12) (x2? — 22 | 
Gi) = #41 See y SOE 4 τ ᾿ 
so that: 


i. = (1 = x) {H + (da 4 82u, 


4 ΕΓ 4 = ee at lll S4uyg +. . | 
+a) -}- cai Ἢ, 87u, + SS +.. | 


᾿ς (3.4.8) 


which is Everett’s formula in central difference form. The reader 
will see, by renaming the forward difference symbols in equation 
3.3.8 that it is identical with equation 3.4.8. 
Bessel’s formula can also be obtained in central difference form 
by means of the operational equation: 
E*u, = F(8) (u, + U9) + G(S)u; 


the form of which is suggested by equation 3.3.7, and where F is again 
an even function of § but G(8) is this time odd. The final result is: 


te = ξ(υ + ty) + (x — ἔ)δωρ + BM(x) (S%u9 + δ΄) + BM(x)B%u, 
4+- BYY(x) (84u, + δέω) + BY(x)8'uy, +... ....(8.4.9) 
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MODIFIED DIFFERENCES 
where the symbols B"(x) are given by: 
Bi(x) = x(x — 1)/2.2! 
Bil(x) = x(x — 4) (x — 1)/3! 
B(x) = (x + 1)x(x — 1)(x — 2)/2.4! 
BY(x) = (x + 1)x(x — #)(x — 1) (x — 2)/5! 
ete. 


.. ++ (3.4.10) 


The notation used here is that suggested by Comrie, loc. cit. 


3.5 MODIFIED DIFFERENCES 


In use, Bessel’s formula is often modified by means of a numerical 
accident. Noting that du, = u, — up, we see that equation 3.4.9 can 
be written, to fourth differences: 


Uz = Uy + xduy + BY(x) (82uy + δῆ) 
+- BM (x)33u, - B(x) (S4u, + δή) .-..{55.} 
Now reference to equation 3.4.10 shows that: 


12 
and a calculation shows that the quadratic coefficient of B!(x) has 
only a limited variation in the range (0 < x < 1). This is shown in 
Figure 3.5.1. 


Bix) os sc(teue) 


Figure 3.5.1.—12 Biv(x) /Bii(x) 
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The limits of variation are between — 2-0 at x = 0, x = 1 and 
— 9.25 at x = 4. It follows that 


B(x) = CB4(x) 


where C varies between — -1666 and — -1875. | 
It was suggested by L. J. Comrie that in many cases it would be 


adequate to write: . 
ΒΒ (ἡ = CB"(x) 


where C is a constant known as the throwback coefficient, so chosen 
as to make the greatest absolute error, £, 


in E = B'*(x) —CB"(x) 
as small as possible. It is fairly easy to see that the required value of C 
is one which makes E,.. = — Emin in the range Ὁ < x <1. To 


determine the value of C, we have from 3.4.10: 


= 1) nr 


We notice that £ is symmetrical about x = } and put z= χ --ἰ so that 


ἊΝ πα 


pa ETP (ε- ἢ 2 4 ἢ (α -- ἢ -- 120] 


(aD) ἐμὰ... ἂν ee 
ia 48 [z 4 ] 

Evidently one of the maximum absolute values of E occurs at z=0 
94 19 

when Καὶ = mad To determine the other maximum absolute 


deviation we put y = z*so that: 

E = dy? — (2 + 12C)y + Gs + 3C)] 
the greatest value of E is given by dE/dy = 0 
i.e. y=%+6C 
whence Ena = — ἐξίθ0 + 1)* 
We thus require 

(60 + 1)* = pgs (Bt 120) 

or 36C2 + 90+ 4=0 
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DIVIDED DIFFERENCES 
whence C=—3+4 V2/24 


To determine the appropriate sign we notice thatC = — 3 + 2/24 
gives a larger absolute value of Εὶ at z=0 than does C=— 3— V 2/24 
so that the appropriate value is C= — 3— ν 2/24 or C2-— - 184. 

Using this result, fourth differences which do not exceed 1000 units 
in the last decimal place are ‘thrown back’ into the second difference 
to give a modified second difference defined by: 


Siilln = δξω, — ο184δέμ, Το συ} 
The Bessel formula is then simply written 
U, == Up + xOuy + B(x) (Siu + Siuy) + BM (x)d uy 


and is then accurate to fourth central differences. 

This idea of throwback can be extended to higher differences than 
the fourth but the algebra for determining the throwback coefficients 
becomes prohibitive. Kopa.™) has given a general method for 
deriving throwback coefficients which depend upon Chebyshev 
polynomials, but it is debatable if the results are superior to those 
which can be obtained by using a least squares approach in which 
C is determined by making 


|, Bede 
0 
a minimum. This technique is discussed in Chapter 10. 


3.6 DIVIDED DIFFERENCES 


When the values of a function are given for values of the argument 
which are not equally spaced, it is still possible to define a system of 
differences. Consider the table: 


* | Sf) 
Xo Ug 
xy uy 
My | Us 
«3 Uy 


The first divided differences are then defined to be: 


Δ΄ = (uy — Up) | (x, — Xp) 
A’u, = (ug — u,)/(*%2 — χα) 


A'u, = (ω,..) — Ur) (ἀκ εὰ — Xr) ...- (3.6.1) 
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Similarly, for second divided differences, 
Al*uy = (A'u, — A’up)/(%2 — Χο) 
A’*u, = (δες — Δ΄) {πᾳ — %) 
εἰς. 
and, in general: 
A’+3y, = (Δ πω — Aug) | (%¢-41 το Χο) 
ΔΎ, — (A thy +4 55 A" un) | (Xnar41 = Xn) nies! e (3.6.2) 


There exists the analogue of the Newton-Gregory formula for 
divided differences: 


lg = Ug + (x το Xo) δ΄ + (x — Xp) (* το %1)A"ttg 
ἐῶ - τὴς — 4)" HL 
(x.— %g)(x -- χὰ)... (ἡ το Hp) Δ τωρ τ .... (3.6.3) 


It can be seen from equations 3.6.1 and 3.6.2 that when the 
intervals x,;, — x, become equal (to 5x, say) the divided difference 
formulae reduce to: 


A’, = Aun |dx 
A! uy, = A%un|1.2. (8x)? 
A! tin = A3utq/1.2.3(8x)? 
ele. 
Δ, = Atty [π|(δχ)" ,νον(8.6.4) 


so that equation 3.6.3, may be written : 


xd A*uy | Au, 
μρ = ty +E x(x — Bx) Sieg + τίν — Bx) (x — 28) STE Ἐ 5 


+ x(x — 5x)(x — 25x)... [x = (r= δι] ee, + ΤῊΣ ἃ 


or 
ly = τίρ + ἔμδιρ + Eph 2g + .. » EMU +... 
where &, is as defined in equation 3.3.2. 


This is the ordinary Newton-Gregory formula of equation 3.3.1 for 
interval dx. 
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3.7 LAGRANGEAN INTERPOLATION 


An alternative formula for interpolation at unequal intervals is due 
to Lagrange. Suppose that, as before, functional values: 


* Τὰ 
Xo Ug 
My uy 
ar | Uy 
*n—-1 Un —, 


are given, and that they are ἡ in number. An interpolation formula 
consists in finding a polynomial, of degree (n — 1), passing through 
the given values. Assume that: 


S (x) = (* — χο)ία τα)... (ἡ — Xn) Σ᾿ -ς ‘csefSurea) 


Put x =x, (s=0,1...n-—1)_ then: 
-.. ___ fies 
(%s — Xo) (χε το %y) « « « (ἅν — %e—1) (%e — %e41) » - © (Xs — ¥n-3) 


= Uy| (%_ — Xo) (%s — χη » « - (χ, — %e—) (%e — Xe41) - - . (Xe — Xn-1) 

sawahedea) 

Equations 3.7.1 and 3.7.2, together, constitute Lagrange’s inter- 
polation formula. 


3.8 CAUTIONS AND PRECAUTIONS 
One example has already been given (section 3.3) of the dangers 
which arise from the blind use of direct interpolation formulae. The 
reverse process, called inverse interpolation, and sufficiently defined 
by the order: ‘given f(x), find x’, is perhaps even more strewn 
a pitfalls for the unwary. As an indication of these, suppose that 
the values: 


* J () 
l 0 
4 l 
16 2 
64 3 
256 4 


are given, and that it is desired to find f(x) when x = 32. (The 
reader will notice that this is really inverse interpolation from the 
table given. previously in section 3.3.) 

The appropriate Lagrange formula is: 
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(x — 1)(x — 16) (x — 64)(x — 256) 

75) = 1. .4--- ἡ — 16) (4 — 64) (4 — 256) 

(x — 1)(x — 4)(x — 64) (x — 256) 
(16 — 1)(16 — 4)(16 — 64)(16 — 256) 

(x — 1)(x — 4)(x — 16)(x — 256) 
(64 — 1)(64 — 4)(64— 16) (64 — 256) 
os (x — 1)(* — 4)(* — 16) (* — 64) 

ὙΦ 56 — 1)(256 — 4)(256 — 16)(256 — 64) 

from which it is easily verified that (132) = — +263, which differs 
considerably from the true value + 2.5! 


+2. 


4+, 


70 
65 
60; 
55] 
να Figure 8.8.1 
45 } 
40} 
ὠμὰ" 
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Legrangean 
| approximation 


Function 


§ 
Poy et ae 
| a, 


ὅσ 40 80120 160 200 240 280 
a 
The reason for this behaviour is quite evident when the mode of 
variation of the function, and of its Lagrangean approximation, are 
considered. The two expressions are shown in Figure 3.8.1. 


28 


CAUTIONS AND PRECAUTIONS 


In this particular example it happens that a linear interpolation, 
in the range concerned, would give a far better result. This empha- 
sizes the fact that, before using an interpolation formula on any 
tabulated function, it should be carefully ascertained that the 
function behaves in a manner capable of representation by the 
formula. 
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NUMERICAL DIFFERENTIATION 
AND INTEGRATION 


4.1 OPERATORS 


Tue differential of a function has already been introduced as D in 
section 3.1. In a similar manner we may introduce an operational 
symbol for integration as: 


\=5( D-?) ΠΕ ἘΠ ν 


Now from equation 3.1.11 we have: 
dxD = log, E = log, (1 + 4). 


whence, expanding: 


ἫΝ a. a 
ὄχ = 4 = 9 3 
or 
eS 4. a A’ : 
D=5,|1 erence = rere ᾿ Ἐν |... -(4.1.2) 
which, when applied to y = f(x) at x = xy gives: 
d ] 
(3), ᾿Ξ §g (Allee) — 243%) + 84am) — J ....(4.1.8) 


In a like manner, raising 4.1.2 to power n. 


sa, a A" A A? n 
D =Gp('-3+5 se] ..««.{4.1.4} 
from which: 
| A? 1] τ 
Dal! -- 4 + 554? -- εὐ. * ) ove of 4,5) 
and 


_. 4 3 
D? = aa -2 44... .) ον (4.1.6) 


These formulae provide a convenient means of obtaining the differ- 
ential coefficients of a function which is given by a table of values, 
and not in analytical form. 
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4.2 CENTRAL DIFFERENCE FORMULAE FOR 
DIFFERENTIATION 


In many applications it is more convergent to have expressions for 
the differential coefficients of a function in terms of central differ- 
ences. The operational method may again be applied in the follow- 
ing manner. 

From equation 3.4.5: 


U(=8xD) = Qsinh-"2 


where ὃ is the central difference operator. 


Unfortunately U is an odd function of 6 so that this equation will 
produce a series of odd powers of 8 for all odd derivatives. Since such 
values of ὃ cannot be obtained directly from tabular entries (see 
the third diagram of section 3.1) a more satisfactory procedure has 
to be found. This can be done by means of the operator yu, defined 
by equation 3.1.10. 


Clearly 
pd = 4(E — E-) CN! (4.2.1) 


pOug = $(uy — U4) 
which can be found directly from a table. 


so that 


Also: 
2u +8 = 2E} ,...«(4.2.2} 
2u --ὃ = 2E-3 nese «[4,3}3} 
whence 
4μ3 — 6? = 4 
or 
p? = 4824+ 1 wes (4.2 5) 
Now we may write: 
uy ames . 
(5) 5 ( a os (4.2.6) 


which will give, for even values of n, an expansion for 1925 in terms of 
even powers of δ. Alternatively: 


] 5) ] (ἢ 
-[-|. =- a+ + (42,7) 
μ (3 μὰ ἐδ ee 
which, by virtue of equation 4.2.5, will give expansions of D” for 
odd values of n, in terms of even powers of ὃ multiplied by μδ' which, 
since n is odd, can be found from tabular entries. 
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To obtain the series we make use of two general results due to 
BICcKLEY®) : 


yee Sn? +22n.. 35n® + 462n% + 1528n ., 
(5) ek a rae 2903040 55 Ὲ 
175n* +- 4620 η8 +- 40724η3 + 119856n 58 
a 1393459200 “4 
385n° + 16940n* + 279884π5 + 2057968n* + 56820482 .,, 
“os 387. 7) θοῦ 
es in ce 
ιν" | n +3... Sn®+52n +135, 
a(z) eh egg ae Tee 
᾿ 35η5 + 777n* + 5749n + 14175 5, 
| 9903040 
4, 175n* + 6720n° + 96794n* + 619776n + 1488375 5, 
1393459200 : 


385η8 -+-22715n* +-536294n* +-6333250n?-+-3740828 In +-88409475 0 
eo oe 1: 
367873228800 
- "" κ5᾿;, ἃ see (4.2.9) 


These results hold for positive and negative values of n. 
From equation 4.2.8 we have: 


(5) = 1-8 + aoe, εν 


which gives: 
ΓΞ iyi 88 — ἠδ deel...) ... (8.10 
Also from equation 4.2.9: 
«(5) m= 1 — 48? ἔοι 88+... 
leading to: 
Ὁ = τ [(μδ) — 38%u8) + ven) — τεοδ'(μδ) +. J 


ἘΠ ΤΟ 
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When applied to a functional value (uo, say) 4.2.11 may be written 
(by virtue of equation 4.2.1) 


Die) = egy Ue — 4-1) — 216% — Bu) 
+ ds(S4uy Su.) —. 2]. (4.2.12) 


which is in a suitable form for tabular use. 
Higher derivatives can be evaluated in a similar manner by the 
use of equations 4.2.8 and 4.2.9. 


4.3 NUMERICAL INTEGRATION 
In many problems it becomes necessary to evaluate the integral of a 
given function between certain limits, and although the derivatives 
can always be found when the analytic form of the function is given, 
the same is not necessarily (or even frequently) true of the integral. 


f(x) 


Figure 4.8.1 


Thus numerical integration, sometimes called ‘mechanical quadra- 
ture’, is a process of importance in its own right, and quite apart 
from its use with tabulated functions. 

The simplest form of numerical integration may be described as 
trapezoidal approximation. Thus, with reference to Figure 4.3.1, it 
is evident that a rough approximation to the integral: 


[toa 
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is given by the area ABCD, i.e. by: 
$(AD -- BC) (x, — Χο) 
= 4(x, — %9)(f/() +/*o) 
Clearly, a better approximation may be obtained by representing 
the arc DEC by means of a quadratic expression in x, and forming 
the integral for the resulting function. 


The quadratic function can easily be obtained from the Newton- 
Gregory formula, and is: 


file) =fl0) + x4f0) +*F >" zepoy ....(4.3.1) 


where /,(x) is the quadratic approximation to f(x), and we have 
taken the intervals AF = FB = |. Equation 4.3.1 can be integrated 
to give: 
2 
[, falx)dx = 2f(0) + 2470) + ἐδ" 0) ....(4.3.) 


or, replacing the differences by their tabular values: 


[, falxddx = AL f(2) + 401) +f(0)] ....(..3.3) 


which is usually known as ‘Simpson’s rule’. 
The extension to the case in which the interval in x is (6x) instead 
of unity is immediate and leads to the formula: 


2(5z) (Sx) 
[ Sala)dx = ᾿ς’ Lf2) + 4f0) Ἐ.Χ0] ....(4.3.4) 


When a function is to be integrated over an extensive range it Is, 
in general, better to add the contributions arising from the applica- 
tion of Simpson’s rule to successive groups of three ordinates, rather 
than to seek an approximating polynomial which represents the 
function over the whole range (see, for example section 3.8). ‘Thus: 


2n(dz) 2(52) 4(3z) 
[πώ σ᾿ fede +] πλάνο... 
0 0 2(6z) 
2n(3z) 
εἰ Salsas 
(2n—2)(5z) 
- 2) 5 f0) + 4) + 22) + 43) Ὁ «Ὁ. Am]... (43.5) 
from equation 4.3.4. This expression is often referred to as the ex- 


tension of Simpson’s rule. It should be noted that the approximating 
function in this case, is not necessarily a smooth curve. 
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A further approximation to the integral of a function, between | 
limits 0 and 3 can be obtained by adding a further term to the 
Newton-Gregory formula (4.3.1). We then obtain: 


felt) =f(0) + 2f(0) - ἈΠ =" 2:0) + ὅκα — 0 = 2,450) 
where /,(x) is a cubic approximation to f(x). Integrating we obtain: 
[-.:π0)άν = 3f(0) + 3400) + 84°/(0) + 849/00)... .(4.3.6) 


or, writing the values of 4, 4? and 48 in tabular form: 


| Fels) = $L/(0) + 3f(1) + 3f(2) +f(3)] .....(4.3.7) 


which is called the ‘three eighths’ rule. 


A more accurate numerical integration procedure is ‘Weddle’s 
rule’, which may be derived as follows. Take the first 6 differences 
and apply the Newton-Gregory expansion: 

. | x—] — I)(* — 2) 
fuls) = f(0) + 2490) + “FY argo) 4 EA 6 Ξ- ὃ gag) 
4. x(x — hs 2) (x — x(x 


3) A*f(0) + Pee) .-- τ 9 A®f(0) 


a(x — 1)... @—5) 


+ 61 S(0) 
Integrating this expression over the range 0 — 6 we obtain: 


[πολ = 60) - 184/(0) + 2742/0) + 24434(0) 
+ ABBAS) + 2845/0) + τὰ Δ. 0) ....(4.3.8) 
Now, with an error of 4*/140, we may write the last term in equa- 


tion 4.3.8 as 4%, 4°/(0) = ,4,4°/(0). The expression then becomes, on 
substituting for the differences in terms of the tabular values: 


[fade = τί + Duty + ug + Gu, + uy + 5u, + lls) ...- (4.3.9) 


which is Weddle’s formula in the usual notation. 


_ The foregoing results are sometimes known as Newton-Cotes 
integration formulae of the closed type. The expression closed is in- 
serted to indicate that the approximating polynomials pass through 
the end points or limits of integration. 
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For the purpose of solving differential equations it is useful to 
have available integration formulae of the Newton-Cotes open type. 
That corresponding to Simpson’s rule can be obtained by integrating 
the quadratic approximation 4.3.1, shifted to operate on x = | as 
base point. Thus: 

felt) = fl) + (x — 1) Af) + Ξ 9 8 -Ξ arg 
and 


[πρᾶν = $1201) — Ὁ) + 2f03)] 


or, for interval (6x) 


4(5z) 
| fels)dx = ἐδ) 201) -- 2) + 2f(3)] -...(4.3.10) 
0 


44 THE EULER=-MACLAURIN EXPANSION 


A general integration formula which is often quoted in numerical 
analysis is that due to Euler and Maclaurin. 


Consider the summation: 


= n—1 
Sfx) = ( ΣΕ") fla) vee. (4:4.1) 
ἄνα α =0 
Proceeding in terms of operators we have: 
a 
cE = TT ...«.(4.4.2} 


Now, by virtue of equation 3.1.11 E = ¢®2) = οὐ and we may 
write: 


“τὰ = ale] woe (44,3) 


and the well-known (3) expansion 


U . U2 Us Us 
pays 2 +B 2! — | a 


ae r+ a 
+ (— 1)"**B, "ὦ μ : Fs 
where B, is the rth Bernoulli number, enables equation 4.4.3 to be 
written: 
l bn J U U 


3 
Foi" 0 309) > ea ne 


vi (4.4.4) 
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The numerical values of the first few Bernoulli numbers are: 
μὸν = i, = B, ἔπεισι ret B, = vo τι —_ $6 
= ves; . = é, Bg — ΕἾΤ B, = a 5k aE By = tt Di 


pi from ager 4.4.4 and 4.4.2 in equation 4.4.1, and 


observing that U = (xD), ὦ a = 5 [ we obtain : 
rma+(n—1)dz 


(@) = 


= (B—1)| 5,[fladde—4fle) +34 δὴ γὼ —F40)47Ma) +... ] 


or 


r=a+(n—1)dz 1 patndz , 
ZF) τ] Sle)ds — Hla + m8) ~ fa) 
+51 (δ) (a Ὁ πδα) —f'(a)] — 32 (64) 9 F(a + nB) —f!(a)] + 


Finally, on rearrangement, we obtain: 


a+néz 
[ fixjdx = [γὼ + f(a + 8x) + 2fla + 28x) +. 
"ἢ + (ἡ — 1)8x) + fla + ndx)] 


“LE (a + n8x) —F'(a)] + 2 Epa + nBs) — f(a) 


δ 2 
— aval ἴα + ndx) —f¥(a)] +... ....(4.4.5) 


which is the normal form of the expansion. 

It may be mentioned that the Euler-Maclaurin formula is of 
little practical use as an integration procedure. It finds an applica- 
tion, however, in the summation of series and, in this connection, 
will receive attention in Chapter 5. 


-ἘΠ 


4.5 CENTRAL DIFFERENCE FORMULAE FOR INTEGRATION 


The Bickley expansions (equations 4.2.8 and 4.2.9) can be used to 
obtain several useful formulae for integration which involve central 
differences, Thus, from equation 4.2.8 with n = — 2 we obtain: 
5\2 
(0) <1 + ἠδ — whe! + snd — (4.5.1 
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which may be written: 
] | 
8°55 (fo) = (8x)" fo + ὐνδλγο — shabYe + τὐλτοδοτο - - -) 
tk ele 
Now putting fy = g’"5 = D*(g,) equation 4.5.2 becomes: 
87g = (5x)*(g'"g + ὑχδ'ξ' — τῆσδ᾽ Ε΄ ὁ το φεσὸ ‘Bo τὸ af ) ὴ 


which expresses the second central difference in terms of the central 
difference of the second derivative at the point in question, and has 
several applications to the solution of differential equations. 


Again, if we put n = — 1 in equation 4.2.9, there results: 
] 
(5)= - ] -- ἰδ + 7200" στ soseoo + ἘῸΝ . (4.5.4) 
or 
l | 
8.5 (fo) = (8x) [ufo — 1'28*(ufo) + 7¥05*(Hfo) — τὸ ξὃσδ'(μ[0) +--+] 
. - (4.5.5) 
Now, replace f, by g’; = D(g;), and observe that: 
8g, = SElg, = (Et — E-#) Ekg, = (Ε — 1)go = δι — 80 
Hs = μΕϊες = ἐ(Εἰ + ΕΠ) Ete, = ἐ(Ε + Nee = = = 6 + 80) 
+e ««(4.59.6) 


whence, equation 4.5.5 becomes: 
— 8 = 3(8x) [g's + δ΄ — v2 (08's + δ΄ε΄ Ὁ) 
+ #5 (8%e', + 94e’o) — εὐξεσ(δ'6΄, + 5%"o) . . .1 --.- (4.5.7) 
This is clearly an integration formula expressing 


a -- (= { ear) 


in terms of the relevant functional values of g’(x). 
Finally we shall consider the expansion of the operator (μδ). 
We have: 


(ud) fo = ἐ(ΕἸἰ + E-t) (Et — E-*) [9 = ἐ(Ε' —E“) ἢ Ξ a —f-) 
4.5.8 
Again, equation 4.5.4 may be written: τοὺ 


μ()-- μι edt + AB -- οὐδλοδο Εν.) 
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or, observing that p? = | + 36? (equation 4.2.5) 


HO = 1 988 — re! bra 45.9) 


It follows that: 
p War Feu) Sf, + 38%, — rhe + ratty — « 


or, writing: ἔς = δ΄, = Dg, 


μδρο = 4(8; — 8-1) = (6x) (g'9 + 3878' oP ost sie - +) 


. (4. 5. 10) 
which is the required result. 


4.6 THE CHEBYSHEV INTEGRATION FORMULAE 


Integration formulae of the Simpson-Weddle type are advantageous 
when used in conjunction with a table of functional values, since 
they make use of equal intervals. On the other hand, each of the 
function values has, in general, to be multiplied by a different 
coefficient. When function values have to be calculated for the 
purpose of the integration, it is often more convenient to use an 
integration formula in which all such values are treated in the 
same manner. 

We thus examine the possibility of finding an integration 
formula: 


[εάν = ἀ{φίχυ) + φίλῳ) +(xs) . .. +8(%n)] + ΚΕ, ....(4.6.1) 


in which k and x, xg, . . . X, are independent of the particular 
function g(x) being integrated. 

Actually, a slightly more general result, given by MuI.ne- 
THomson (8) will be examined: 


[ΠΡΟ κοῦ ἀν = Lets) + 86s) +. +8) +Ry ....(4.6) 
where k, x1, %_) «0. ἀμ depend on F(x) but not on g(x). 


First, assume that g(x) can be expanded in a Maclaurin series: 


g(x) = ((0) -Ε χφ 0) +3-,2"(0) +-. FeO + Gee (8) 


. (4.6.3) 
in which the remainder term has 0 < ἔ < x. 
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We then have: 


+1 +1 | | x2 
| Fla)e(a)ax = | Fe [ 00) + χε) +5480) + - 
..1 ἘΠ 


+ ~ Ὁ) ἘΠ Ἦν er) |x 


τ ΣΟ F(x): mart [ἃ ἜΤ ΤῸ a aR(a) de ον ν (4.6.4) 


Again, using equation 4.6.3 with « = %,, x, . .. Χρὶ 
k [g(%s) + (χω) + - - - +. 8(%n)] = kng(0) + ἀφ'(0) 2x, 


kg"(0) 5 ata rane Σ ΧΡ nt+l(g ) 


n! 1 
ω (463) 


Whence, if equations 4.6.4 and 4.6.5 are to be ae to the 
error specified by R, in equation 4.6.2, we have, on equating 
coefficients of g(0), g’(0) . . . g"(0): 


kn = [" F(2)dx 


ia ead see + 
: fal 


k Bx, = ["  xF(x)dx 
r=] -1 
k Det = [* <8F(x)de erat 


r=] 


k Bx” = ie x™F(x)dx 


r=] 
Together with: 


+1 ἢ 
R= | 4 fis fo UE τη" ΝΡ = 
.. (4.6.7) 


where O<écxand0<ceé<xz(r=1 .. η). 
To determine the values of x,, x, . . . ἀμ from equation 4.6.6 we 
may adopt the following procedure. 
First let: 
2x” = Sm ....(4.6.8) 


r=l 
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1 
ip εἰ "x F(x)dx 
so that im Ξε 1] xF(x)dx = -ττ τ ---- ..(4.6.9) 
-1 οἴ ρ)άχ 


The quantities s,, are thus the sums of the mth powers of the n 
quantities x,, *,, . . . ἀρ which suggests the roots of an equation of 
degree n. Assume then that x, x, . . . x, are the roots of: 


Sn(*) = x? + aye? + agxe®- 27 +... +a =0 ....(4.6.10) 
so that: x te ax®—! + age®—* 2. . + ay 
= (x —x,)(x —x,) ... (x — χ) «... (4.6.11) 
Now put x = |/y so that, multiplying both sides by y" equation 
4.6.11 becomes: 
L + ay + ay... + any” = (1 — %,y)(1 — ayy). .. (1 — Xn) 
Taking logs of both sides: 


πω 
loge(1 + ay + @,y? . .. α,}"} = 2 log.(1—x,9) 


os ae τὰ 
= BS bang ἜΝ gee ἘΥΡ bre) 


5 5 
sleet, © died )).-τὖῷὟ ek ek Ξ nT - -. κ" κα 
. (4.6.12) 
on expansion and by virtue of equation 4.6.8. 
Next take exponentials of both sides: 


l+aqy+ay?+... +49" 


= exp [— 51.9 — (52/2)? — (s5/3)9* . — (εἰ = = 1 
«ἃ 4.6. 13) 


So that, upon expanding the exponential, and equating conduit 


we can determine the coefficients a), in terms of the power sums Sy. 
The first few are: 


ἃ) ΞΞ - δ 
a, = — 45, + 45,’ 
ἄς = — ἐς: + 45452 — 95," 
ας = — ἧς, + 95453 — ζηγῆες + ἔπε + a5! 
Gs = — $55 + ἔτος + B5a53 — 51°53 — δε + 1'251°52 το T2071? 
ἄς = — 356 + $5155 + 35054 + ΣΥΝ 51°54 — ΩΣ 
— Pa5e8 + 755755 + υεηῖο — ἐξοχῆς, + ato 
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Thus, knowing the form of F(x) and using equation 4.6.9 to deter- 
mine the values of s,,, we can write down the equation 4.6.10 and, 
by solving it, obtain the values of x, . . . Xp. 

The most usual technique is to take F(x) = 1, we thus obtain the 
following equations /,(x) = 0, and the relevant roots for insertion 


in 4.6.2. 

n | k(=2j/n) | Jn(x) : Roots 

2 l xt— } + "5773 503 

3 ἃ x(x* — 4) 0, + -7071 068 

+ + xt — gx? + γε + +1875 925 + -7946 545 

5 ᾿ x(x* — §x? - 7.) 0, + 5745 414 + -8324 975 

6 4 x? — x* + dx? — 4h, | 4 +2666 353 + +4225 186, + -8662 476 


It should be noticed that the remainder, after an n point integra- 
tion using the Chebyshev method, is zero if g(x) is a polynomial of 
degree not greater than n. 

The Chebyshev method is particularly suited to use on an auto- 
matic digital computor since, in such machines, the fewer different 
kinds of arithmetic operations which have to be performed to arrive 
at a given result the better. 


[An alternative method of approach, which gives the polynomials 
Jn(x) directly, is as follows: 


Let 
μῷ = ἃ —m)(2 το 4%)... (ὦ το χρὴ) =0 


foe) =2* (1-3) (1-9). .(1—*) 


n | 
= 2" exp 2 log (1 - 5) 
rol 


then: 


Since zn is finite, we may interchange the summation symbols so that: 
Σὰ 
Sn(2) = 2" exp — Σ(Ζ) 
but, by equation 4.6.9, 
. +1 +1 
axf = n| x F(s)ae [] F(x)dx 
r=1 -1 —i 
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whence, 


fale) = 2 exp — En)” xt Reds | wt |” Roa 
= z" exp nf Fes -5* 7] { ΓΝ 


=e exp [δῶν iy *) ἂν / J Fede 


where f (z) is taken to include only positive powers of z (including 
zero) in the expansion. When F(x) = 1 this becomes very simply: 


In(2) = 2" exp — ἽΕΣΞ Tis 2 Ὁ. ) 


from which the preceding values of f(x) can be quickly derived by 
expansion. | 

It may be noted, in conclusion, that the Chebyshev integration 
formula becomes unsatisfactory for n = 8 and for n > 9 since, at this 
point, the values of x, are no longer restricted to the range (— 1, + 1) 
and, in fact, may not even be real. 


4.7 THE GAUSSIAN INTEGRATION FORMULA 


The Chebyshev method of approximate integration is one of a 
family of such results which may be written: 


δ 
J), F(#) -g(x)de = το φίδι) + τον. β( χω) «αν Ἔ wag n) + Rn 
. (4.7.1) 
where the weight functions* (w,) and the associated points (x,) are 
independent of the particular function g(x) which is the subject 
of the integration. [They depend, however, on F(x).] 

In section 4.6 we investigated the way in which the values (x,) 
had to be chosen in order to make the weight functions equal. 
We shall now consider the problem in slightly more general terms. 
First take F(x) = 1 and assume that g(x) can be expanded in a 
convergent power series in x so that: 

a(x) = ag +t axt... tage" +... 1.6. (47.2) 
* Sometimes known as Christoffel numbers. 
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Next, take any set of n points x, . . . x, and set up the Lagrangean 
approximation to g(x) (see section 3.7). This may be written: 


L(x) = πῷ 5 ier Fone eee (4.7.3) 


where 
0) =: (x — x,)(% το χῷ) ... (χ το χ) ....(4.7.4) 


and Π' (χ) is used to represent ἀ Π|ἀκχ. 
fi n 


Now L(x) coincides in value with the function g(x) at each of the 
points (x,) (r= 1... πὶ so that we may write: 


g(s) — L(x) = Is)(bo + be +... bat...) 


where the series on the right is again assumed to be convergent. We 
thus obtain: 


[τὰ = -ἰ L(x)dx +f (11(x) .2 dat) dx 


- Seale) 1. ie sae o(4c7.5) 
rl 
where 
ν Gide | 
il, -\ gare) . ..+(4.7.6) 
and 
ὃ oO 
-| (11(3)2 bat) ....(.7.} 


Thus, if g(x) is a polynomial, of maximum degree n, we see that 
R, = 0, and that the n values (x,) can be chosen arbitrarily so long 
as we choose w, as defined by equation 4.7.6. 

Since this situation is equivalent to saying that there are arbitrary 
constants available, it is reasonable to enquire if these can be deter- 
mined so as to make equation 4.7.5, with R, = 0, true when g(x) is 
any polynomial of degree (2n — 1). 

Thus we require 


Ry -{[ ME “bat Jax = 0 


διαῦ 
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for arbitrary ὃ, (s <n — 1), and this can only be true if: 


[πω war τος =0, BE icce]) 78) 


a ἢ 


Now it is well known that the Legendre polynomials, which are 
conveniently defined for this purpose by means of Rodrigues 
formula) 

d" 


Pols) = 1, Pal®) = o5 7 gin ἃ 


x21)" ....(4.7.9) 


are such that (): 
(Pals) xtdx = 0 


for all integer values of s less than n; furthermore they have n distinct 
real roots in the interval (— 1, + 1). 


The integral (4.7.8) can be transformed by means of the sub- 
stitution: 


A b 
i= “τ “τ 


to take the limits (—1, + 1), and it is thus seen that if, after the 
transformation, we take the values of (y,) to be the roots of 
P,(y) =0 the resulting 2 point quadrature formula will be exact 
for all polynomials g(x) whose degree is less than (2n). 

An identical argument can be carried out for the more general 
integral (4.7.1). If this is assumed to be transformed to limits 
(— 1, + 1) we have: 


[° Flx)g(x)de ταν σίου) + wa glee) ++» «+ wngltn) + Rp 


. . (4.7.10) 
where: πω 
+1 x 
F(x;) n 
τ = IT' (x;) in (x on Χο) . . (4.7.11) 
and 
R, = I] xb ey Lv i 
(a0 Σ.»»)ὦ (4.7.12) 


(x) (7 = 1. . . πὴ being the roots of P,(x) = 0. 


The values of the roots (x,), and of the corresponding weight 
factors w,, for the important case F(x) = 1 are given in Table 
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4.7.1. They are taken from the basic table of Lowan, Davins and 
Levenson (85) who give values to 15 decimal places for values of n 
up to 16. 

Table 4.7.1. Gaussian integration points and coefficients 


n= | 
x,=0 w, = 2 
n=2 
x, = — x, = *57735 02692 Ξ ἴθ Ξε w, = | 
i= 
x, = —, = .7 7458 66692 wy = Wy = ἢ 
x, τα Ὁ Ww, = ἢ 
n== 4 
x, = — x, = *86113 63116 w, = w, = "34785 48451 
x, = — xs = -33998 10436 ‘ Wy = wy = *65214 51549 
A= 
Χῃ = — x, = 53846 93101 WwW, = w, = *47862 86705 
x,= 0 5 w, = "56888 88889 
i= 
x, = — x, = 93246 95142 w, = w, = 17132 44924 
χε = — x, = *66120 93865 tw, = w, = «36076 15730 
Χὰ = — x, = ‘23861 91861 P W, = τῷ, = -46791 39346 
a= 
x, = — x, = -94910 79123 w, = wy = + 12948 49662 
Xp = — xg = *74153 11856 W, = w, = -27970 53915 
Χὰ = — xg = "40584 51514 ἔθ = τὸς = *38183 00505 
i= . w, = +41795 91837 
r= 
xX, = — x, = *96028 98565 w, = w, = *10122 85363 
x, = — x, = *79666 64774 δ. = w, = *22238 10345 
Ny = — xy = *52553 24099 W, = w, = *31370 66459 
x,= — χε = "18343 46425 ὃ w, = τὸς = "36268 37834 
n= Q 
x, = — x, = *96816 02395 w, = ws = *08127 43884 
x, = — x, = °83603 11073 w, = w, = 18064 81607 
Xs = — x, = ‘61337 14327 Ww, = w, = +2606] 06964 
xy = — x, = °32425 34234 w, = we = *31234 70770 
x, = 0 - ws, = "33023 93550 
=> 
X= — x, = *86506 33667 W, = wy = *14985 13492 
λα = — xg = *67940 95683 w, = w, = -21908 63625 
X= — χη = *43339 53941 WwW, = w, = -26926 67193 
Xs = — xg = "4887 43399 Ws, = τες = 29552 42247 


fh 
It may be noted that, 2 w,= 2—a result which is obtained by 


rel 
taking F(x) = g(x) = 1 in equation 4.7.10. 

The Gaussian integration method just described may be extended 
to other ranges than ( — 1, +1). Probably the most important 
from the practical point of view are those ranges in which either, 
or both of (a, 6) are infinite. Finiteness of the resulting integral 
implies that, if g(x) is a polynomial, F(x) is not, and it is natural 
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to relate the extended method to particular forms of F(x). Thus 
consider the integral: 


e-*g(x)dx ....(4.7.13) 


where g(x) is again expressed by equation 4.7.2. We once more 
seek an integration formula of the type shown in equation 4.7.1 
and set up a Lagrangean approximation, but this time to g(x) 
only. If: 


g(s) — L(x) = Πρ) +s ἘΠ. ον bbe...) 


we obtain, from equation 4.7.13 


fe e(x)dx =| e-*L(s)ae +f | 7 (x) Σ be" | rds 


"0 0 

= Σ τυνσίχρ) + Ry +++ (4.7.14) 

where 

1 ο €~ "IT (x) 
ccperiel! aia ον 4.7.15 
τ Tre), ea we 
and 
R, -| | Πρὸ Σ byt ἢ τὰκ ....(4.7.16) 
0 n g=( 


If we wish equation 4.7.14 to be exact for all polynomials g(x) of 
degree less than (2n — 1) we thus require: 


oO 
Ι II(x).x8.e-7dx = 0 (s=0,1...2-—1) 
0 th 

Now in this case the Laguerre polynomials defined by: 


L(x) = δ = (nS) “οὐ 01 ἢ 


are appropriate since it can be shown (7) that: 
Ε e-*x8L,,(x)dx = 0 
0 


whenever s < n, and we thus take the sampling points (x,) to be the 
roots of L,,(x) =0. Some weights and coefficients are given) in 
Table 4.7.2, 
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H,,(x) = (— 1)" τ (ε΄ “Ὁ (4.7.18) and the (x,) are the roots of T,,(x) = 0. 


for which it can be shown that: 


NUMERICAL DIFFERENTIATION AND INTEGRATION | THE GAUSSIAN INTEGRATION FORMULA 
Table 4.7.2. Laguerre polynomial roots and weight coefficients | In a like manner the Chebyshev polynomials: 
n= | 
= | w, = Ϊ 
Ξ ΞΕ τὼ Ty (2) = 554 608 (n οο5-:} x) .. . » (4.7.20) | 
᾿ς tame 2 mts 
a n= 3 + as | have the property (10): | 
x,= 0-41577 46 w, = *71109 30 
X= 229428 04 w, = *27851 77 | +1 AT, (x) 
χε τε 6-28994 51 w, = "01038 93 | =< ας τα 
n= 4 ..1 V1 — x? 
x, = 0-32254 77 w, = -60315 41 
<o ; ἜΡΟΝ ἐξ τόρ δε, fee Ἢ | for s < n which leads at once to: 
x, = 9-39507 09 w, = -00053 93 
| n=5 _g(x)dx κΣ 
x= 0-26356 03 w, = +52175 56 a = 2 απ πῶ νΝ 
% = 1.41340 3] τὸς = +39866 68 Y= ἢ μὰ 
x,= 3-59642 58 tw, = -07594 24 
x,= 7-08581 00 w,= -00361 18 where 
x, = 12-64080 08 w, = *00002 34 
ι Γ" I(x) 
Again, if the integral required is of the form: τυ = Wes | —--_ *_____—_. , dx 
Ξ 3 er q . IT (x,) J —] (x — %)V 1 =a x" 
+0 n 
Ϊ “πο ρ(α)άχ 
A J) Lae 2 lz 
it is appropriate to consider the Hermite polynomials) defined by: eS i Ns ἘΠῚ 


Table 4.7.3. Roots and weight coefficients for (I— x*) Ρ᾿() 


+c | 
\ oF AH (x)de =: n= 1 " ; | 
ne ἃ | 
whenever s < n. These lead to the formula oes x= : θεν ἔν δὲ ΙΝ 
| ΠῚ ΞΞΞ 2 = : 
st ar 0- 16666 67 
ες τ' Χὰ ἘΞ -α χὰ ἴθ) ΞΞ τῦς ΞΞ 
| " ὠὰ Σ ως ee HELD) ee ri τὴ = w, = 0-83333 33 
an n= 4 
where x1 =—x,=1 δὴ ποῦνε ὃς δ ee ὧν 
--“«" X, = — x, = 0-65465 37 R= w= 1111 
l Ὧν "ΠῚ ἃ χὰ “wa 0 wo  ~O-71il! i! 
= π π-- een CLs n= 5 : 
IT (Xr) J ὦ ὦ — χὴ ree ἩΠῈ 10, = tH, = 0°66666 67 
7 χς = — x, = 0-76505 53 w, = w, = 037847 50 
+00 on 5 = — x, = 0-08135 70 WwW, = w, = 0: -55485 84 
om -z" n=6 
a | ue) ‘, so a — ἀν, ΞΞ w, = w, = 0-04761 90 
ἘΣ = went ' X= -- χε τε 0. 83022 89 Wy = ἴῦᾳ = 0.27682 ° 
χ = — x, = 0:46884 W, = w, = Ὁ" 
and (x,) are the roots of H,(x) = 0. = "as uw, = 048761 90 
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To conclude this section it may be mentioned that it is sometimes 
desirable to have an integration formula of Gaussian type, but in- 
volving the function values at the limits of the range of integration 
(a, 6). Such results are usually known as ‘Lobatto’s or Radau’s 
formulae’ and a typical example results from the use of the poly- 
nomial: 


(1 — x?)P’,(x) .«...(4.7.22) 


instead of equation 4.7.9. Formulae based on equation 4.7.22 are 
less accurate than the true Gaussian results, being exact only for 
polynomials of degree (2n — 1) when (n +- 1) points are used. The 
chief application is to the integration of functions having the value 
zero at both ends of the range of integration. 

Roots and multiples for the polynomials 4.7.22 are given™) in 
Table 4.7.3. 


4.8 OSCILLATING INTEGRANDS 


A class of integrals which is of practical importance is that typified by: 
6 sin 
|S?) cos δ) 


For finite values of a and ὁ Filon’s method'!? is very suitable. 
Here, the range is divided into 2n equal parts so that 


b =a + 2nh 
and, if θ -- kx 
it can be shown that: 


[Χρὴ sin (xp) dp = AL — af f(b) cos xb — f(a) cos xa} 


+ BSas + ySys_;] 


[νῳ cos (xp) ἀῤ = ἡ[α f(b) sin xb —_f(a) sin xa} 
+ BCs + yCos_ 1] 
a, B and y are defined by: 


a = py (0 + @sin @ cos θ -- 2 sin? 6] 

5 = [@(1 + cos? @) — 2 sin @ cos 6] 
= 4 = 

γ [sin 6 — @ cos 6] 
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Sys and Cy, are the sums of all even ordinates of the curves 
y =f (Pp) sb (xp) between a and 6 inclusive less one-half of the first 
and last ordinates. S.,_, and (ὼς...) are the sums of all odd ordinates. 
Tables of a, β and y have been produced by Filon. 

For an infinite upper limit, the method of choice is to use: 


|. £0) sin x dx = ( — 1)"(1 + a,6* + a,64+ ... )f(nz) 


where the a; are the coefficients (δῆ) in the expansion of 
(1 + (log E)?/m?)— and: 


a, = — 0-10132 118 
a, = + 0-01870 941 
a, = — 0-00387 695 
a, — + 0-00084 579 


ΕἾ. 


4.9 ERRORS OF FINITE DIFFERENCE APPROXIMATION 


Except in the case of the Chebyshev formula, we have not attempted 
to estimate the errors produced by the approximate integration 
formulae. A general method of arriving at such error estimates has 
been given by ΜΠ ΝΕ 13), and we shall now indicate its application 
by considering the error produced by Simpson’s rule (4.3.4) and by 
its Newton-Cotes open variant (4.3.10). These particular examples 
have been chosen because the error estimates will be needed later, 
in Chapter 6. 

In the first place, we make the observation that all finite difference 
formulae are exact when applied to any polynomial whose degree, n, 
does not exceed a certain upper bound, N, say, which is a constant 
for the particular formula involved. It follows that if the error is to 
be estimated when the finite difference formula is applied to an 
arbitrary function, a suitable technique will be to expand that 
function in a Maclaurin series: 


Sf) =f) + FF) + a pO) +... + - po + 
cep n+1(€) εν. (4.3.1) 


where 0 < é < x. 
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Suppose that the actual operation to be performed is O and that the 
finite difference process is-represented by, F; the error, or remainder 
R, is then given by: 


R = ΟἹ f(x)] — FI f(s] εν. .(4.3.2) 


Now, if in equation 4.9.1 we make n = N, where N is the highest 
degree polynomial for which F is exact, we may write 4.9.2 as: 


aN 
Ry = οἱ τ ΤΠ song | = ΩΣ | ....(4.9.3} 


(Since O and F are assumed to be linear operators, so that 


Of tSe+ -- fn) = O(ft) + O(ft) +... + O(f,) ete.) 


whence, replacing f("+)(£) by its maximum value in the interval: 


Ry < CEST FEE {cert — erty) ....4.9.4) 


We now apply this result to estimate the error produced by 
Simpson’s rule. We know that the result is exact for quadratic 
polynomials, let us therefore try N = 2 in equation 4.9.4. 


2(8z 
O -| a 4(8x)* 
0 


F =) ya) + 40) +0] = © (280)? + “(8.51 = 4(δ.) 


whence the rule is also exact for cubic polynomials, 
Next put V = 3 


bz 5 
0 -| xidx = 32 Se 


0 
5 
F = ©) 1(28x)¢ + 4(82)4] = 20 
Whence, from equation 4.9.4: 


l | dx)§ 
Ry = 41 a (3, _ af) (dx)5 =— oll ae . (4.9.5) 
which is the required estimate. 


Next consider the Newton-Cotes formula (4.3.10). 
Here again it is readily shown that for cubic polynomials the 
formula is exact. For N = 3 we obtain: 
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4(52) Δ δ 
0 


F = $(8x) [2(38x)* — (28x)4 + 2(8x)4] = #82(8x)5. 
Whence, from equation 4.9.4. ; 


Ry < 7 C$) (2924 — 592) (8x)5 — 28(8x)5 6%)... (4.9.6) 


so that the potential error is 28 times that of the Simpson’s rule 
formula. 
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5 
THE SUMMATION OF SERIES 


5.1 SOME GENERAL OBSERVATIONS 


ALTHOUGH the subject of this chapter might be thought properly to 
belong to algebra, it is unfortunately true that many recent books 
on this subject give little practical guidance on appropriate methods 
of summing series numerically. It is therefore proposed, in this short 
chapter, to give one or two applications of finite difference operator 
calculus to this problem. 

In the first place we may observe that many series have terms 
which proceed in unit steps of some argument, for example: 


(x) - Σψυ, x) sivas taeded) 


where r takes the values 0, 1,2... 


An alternative method of writing equation 5.1.1 makes use of the 
operator £ and is: 


ta) = (£2) (0, x) «oe e(5e1.2) 


In this form the ‘sum’ may readily be found from the well-known 
formula for the geometric ep ierire thus: 


¢(x) = Sari ....(5.1.3) 


Suppose now that y (r, x) can be represented as the first difference 
of some new function £(r, x) so that 


P(r, x) = Ag(r, x) 
equation 5.1.3 now becomes: 
Ἐπεὶ ] 
φ0) =~, ARO, x) 
or, observing that EF — 1 = J, 
f(x) = (ΕἼΤ — 1)8(0, x) 


whence: 
p(x) = €(n + 1, x) — €(0, x) vee (5.1.4) 
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which has thus summed the original series—at least formally. It 
must, unfortunately, be remarked here that only a limited number 
of functions exist which are recognizable as first differences of other 
recognizable functions, We shall consider two classes of these in the 
next section, 


5.2 DIFFERENCE FUNCTIONS 


An elementary example of a function which can be expressed as the 
difference of values of a known function has already appeared in 
section 3.2, equations 3.2.7 and 3.2.8, Thus the factorial function 
x) can always be written as a first difference: 
χιπν 1) 
x™) — Δ ------- rer θιν 
(m+ 1) — 


and, since any polynomial in x can be expressed as a polynomial in 
factorial functions, it follows that any series whose terms are poly- 
nomials in x can also be summed. 


For example: 


Σεϑ = Σ 2(z — 1)(2 — 2) + 32(2 — 1) +2 


zu) z=0 
== 23) 4. 3.2.2) 4. 26) 


_ Gan 


(2) 
pone, Stl 


by virtue of equations 5.1.4 and 5.2.1 and the fact that 0© = 
Or, by expanding and simplifying: 


be- [ep] 


which is a well-known result. 
Again: 
A(a*) = (a(#+8) — g*) = (a®* — 1)a? “ας. {85.68.5} 
so that: 


¢=4(5—] 2. (5.2.3) 


a relation which enables any series which can be put into the form 
2ca* to be summed. 
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9.3 THE EULER TRANSFORMATION 
Although the actual result of equation 5.1.4 is of limited utility, 
the analysis which leads to it can be applied to the summation of 
slowly convergent series with a considerable improvement in rate 
of convergence. 
Thus, consider the series of alternating terms: 


Sy = Ug — Uy + Ug — Us + -.-΄|'Δ τῳ +(- ate ἴη-- πο τ ν 8} 
This may, by the preceding analysis, be written: 


1 + E* 
Sn = I+E Up εν 5.5} 
Now if u, — 0 as n > 00 we may assume that: 
" 
δ΄ Ξε ἴδ, Ξε --ἷ- 
and since Ε-Ξξ 1 -- 4 
" 

S=4 = 

] Ἔν 


whence, expanding: 
S=4 [uy —34u, + 34%, —44%u,+...)] ....(5.3.3) 
which is usually known as the Euler transformation. 


As an instance of the power of this transformation we may men- 

tion the example given ri BroMwIcH (0) who considers the sum: 
l ] 

5 TB “π᾿ “JE * Os 
and shows that, by taking the first 6 terms of this series and evalu- 
ating the differences from the next 7 terms (i.e. up to and including 
1/4/12), the sum is obtained as 8) = 0-6049 correct to 4 decimal 
places. To obtain the same accuracy over 105 terms of the original 
series would he needed! Practical points in the use of the Euler 
transformation are that the first few terms of the original series 
should be added separately and the transformation applied to the 
remainder. By taking two different starting points, in this manner, 
a check on the results of the calculation and an estimate of the 
accuracy can be had. 

One of a number of extensions of the Euler transformation, 
suggested by Tomlinson Fort, is particularly applicable to an 
oscillatory series in which up, 4,/u, <= B-. (B > 1) 
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APPLICATION OF THE EULER-MACLAURIN FORMULA 


We put u, = (8—1)"v, in the original series and thus obtain: 


te ee ἢ | ae: Mi. Ὁ 
oe Abe ae (1+ οἱ :) 
1+, 
or 
ὡς ἢ, Lae Ἢ 
~ 145 ΤΈβ (epee ἜΝ | 


—. 


Since the series in v has terms which are all nearly satel the 
difference series (5.3.4) converges rapidly. 


5.4 APPLICATION OF THE EULER-MACLAURIN FORMULA 


It is evident that the Euler-Maclaurin formula, equation 4.4.5, can 
be rearranged to give: 


a+ndz 
2 fla + 8s) = ga) fled + ALle) Ἔα + ms) 


+ [{΄(α + dx) —f’(a)] — i Cfili(a + ndx) — fi(a)) 
Sx 
+ SP Lp(a + ns) —f"(@)] — ᾿ς, “νι. (6.4.1) 


or if, as is usual 8x = 1, ἀπά ἃ ξε ῦ 
Efe) = [" fls)ax + HO) +f)] + ALS”) -- FO] 
— rol fn) — F™O)] + sabes ΓΤ — FOV] — ....(5.4.) 


If, in addition, f(n) -> 0 as noo for all s, the series converges 
and: 


ΣΤῊ = |°flayax + 1/00) — ae f'(0) + sboS™ (0) — τοξτο Ὁ) 
7 54. capa 


These formulae are suitable for application to series of slowly 
Convergent positive terms, but are also useful analytically for the 
algebraic summation of series. Thus, if f(x) is zero for all s greater 
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than some lower limit, equation 5.4.3 will give the sum of the 


co 
infinite series 2 f(r) in finite terms. 
r=0 
_ Any polynomial series can be summed by means of equation 
5.4.2, To take the previously discussed example 
n 
ar =|" r'dr +- 4n®° + 75.3n? — τὰς (6 — 6) 
r=) 0 
πὸ πὸ nt [πίῃ Ὁ 1)1: 
a ae τὰν 2 


in agreement with the result given in section 5.2. 
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6 
THE SOLUTION OF ORDINARY 
DIFFERENTIAL EQUATIONS 


6.1 INTRODUCTION 


THE numerical solution of a differential equation, ordinary or 
partial, is a problem which has engaged the attention of mathe- 
maticians for many years. Numerous methods have been proposed, 
and some have actually received practical trial; on the whole, 
however, there appears to be no authoritative statement as to a 
best method, or even a set of such recommended methods. 

Upon examining the literature we find, for example, that 
WHITTAKER and Rosrnson@) give only one method of solution 
(that of Bashforth and Adams) and state that this is the best. On 
the other hand, Harrree‘*?) makes no mention of this method at 
all, and a like remark is true of Mitne(®, although, in a more 
recent publication‘), he points out that most methods are in the 
nature of modifications of the Bashforth-Adams process. 

Ordinary differential equations may be divided roughly into 
two types for the purpose of numerical solution, depending upon the 
form of the boundary conditions to be satisfied by solution. The first 
class of boundary condition may be termed ‘one-point,’ by which 
it is to be understood that all of the conditions have to be satisfied 
at a particular point (x,y), say. The second class of boundary condi- 
tion is a ‘two-point’ one in which, for example, the function value 
may be given at one end of the range of integration, and the deri- 
vative may be given at another. Of course, with equations of order 
higher than the second ‘multi-point’ boundary conditions are pos- 
sible; we shall not, however, be concerned with these in the present 
work. It may be mentioned that the one-point condition is some- 
times said to lead to a ‘marching’ problem, and the two-point 
condition to a ‘jury’ problem, for reasons which will become clear 
later. 

In this chapter we shall be concerned chiefly with one-point 
problems and shall mention the more complicated two-point 
variant only in so far as it can be solved by essentially one- 
point methods. A.more satisfactory approach will be given 
later, in Chapter 8, section 8.6, when relaxation solutions are 
discussed. 
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6.2 INITIAL VALUES 


In many of the finite difference methods of solution it is necessary 
to have one or more values of the solution, and possibly of its 
derivatives, near to the starting point. These are usually best 
obtained by a process rather different from that used in the re- 
mainder of the solution and we shall give, briefly, the two methods 
most frequently used. 

The first method is that of Picard, and depends upon the 
‘guessing’ of an initial solution. Consider the first order equation: 


dy | 

ἃς - 62 s+ (6.2.1) 
and assume that near to the boundary, (a, δ) say, we can guess an 
approximation: y = g,(x) . +++ (6.2.2) 


We then substitute in equation 6.2.1 and integrate to obtain: 


y=b+)" flea] [= δε), say] 
the process is now repeated: 


y=b+ |" Sls gx] = 89%) and so on. 


This process, if it converges, will give an approximation of any 
desired degree of accuracy near to the initial point. 

The range of application of this method is fairly limited, since 
the approximating functions /[x, g,(x)] have to be integrable in 
closed form. As an example of the use of the method consider the 


= i gm 
equation : = = x? + 2xy with x = 0 when » = 0. Take: 


a(x) = δ 
then r 
2x4 x* . Ds 
= 2 <a = pe τος ες 
ξε(χ) | (- + 3 ) 7a 3 3.5 
ο 2x4 = 4x8 χϑ Dx5 4x7 
x)=| (x? 4+ —+4+— =+s5+5-= 
éa(*) J ( eo ἘΠῚ 83 1 8.5 5.5} 
. 2x4 = 4x8 8x8 
= RM ἢ. epee πο τον ioe 
84(+) | (- +3 +55t+ge5)% 
= 2x5 4x7 8x*® 
3 13557135.7+35.709 ~ 
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The reader will easily convince himself that the expression agrees 
with the analytic solution: 


ya-btie | ἐπ τ᾿ dex 


when the latter is expanded in a series. It is also worth noticing that 
the Picard solution, in the example, is much easier to compute than 
that derived from the analytic solution. 

The second method of obtaining an approximate initial solution 
is by means of a Taylor series expansion. Thus for y = ὁ at x =a 
we have: 


nn -- a xy — n 
y= bt FD g(a) FOV γὼ τ... + FM may -... 
.... .((6.2.3) 

and, from the differential equation: 

> a 

τ᾿ = 8(*) =f) 1. + (6.2.4) 
giving: g’(a) = f(a, δ) εν... {6.2.5} 
Differentiating 6.2.4 and substituting the boundary condition and 
6.2.5: 

ἀν Of(x,y) 8{5,}} dy 


aa ake Ἢ ὃν ‘dx 
a Ν ἢ 
oof] + [2] ston 
=O Sens ἐπ 


δ 1 


and similarly for derivatives of higher order. 

The process may be illustrated by our previous example, Since 
x τεῦ, y=0, the appropriate expansion (really a Maclaurin 
series) is: 

x* x8 
y = xg (0) + 548°) +3580) +--- 

The differential equation, together with x = 0, y = 0 gives: 

δ΄ (0) = [x* + 227], o = 0 

g" (0) = [2x + 2y + 2x9], 9 = 0 

g) (0) = [2 + 2γ' + 2γ' + 2xy"]o, 0 = 2 

g 4) (0) = [49" + 2»" + 29 ]o, o = 0 

g) (0) = [6 + Ay) +4 2χν M]o, 9 = 16 

g (0) = [8 + 24M + 2χγ(5)}., 9. = 0 

g (0) = [10γ(8) + Ay + 2xy)] 6, 9 = 12.16 etc. 
which will be seen to agree with the series previously deduced. 
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It will be noticed that the Taylor series method, in this example, 
involves slightly more work than the Picard technique; on the other 
hand, differentiations involved can always be performed, for the 
commonly encountered functions at least, whereas the integrations 
of the latter method can only be carried out for relatively simple 
functions. 

. On the other hand, it is possible to carry out the Picard integra- 
tions numerically, in the required range, and this is often a simple 
and rapid procedure. Certain integration formulae are required for 
this operation, and these are obtainable by the same technique as 
was used in section 4.3. A useful set of results is the following: 


(62) 
| fix)dx = © pa51y(0) + 646/(1) — 264,2) + 106/73) — 19/(4)] 
0 
+ Tao Δ 
2(δ1) | (8x) 
[ fisyax =F t29t0) + 124/11) + 2472) + 4713) —f14)) 
0 
+ Tao SS 
a 
J flayax = fargo) + 102/(1) + 73/12) + 42913) — 3714) 
0 
+ Tao SS 
es 4(5x) , | 
| fla)ax =“? (70) + 3201) + 19/2) + 99/3) + 114) 
B(8x)" 
— “945 Sm 
janes 


where Edie is some value of f™ in the interval (0, 4). These are 

readily obtained by integrating the approximating polynomial: 
x(x — | (x — ΒΕ: 

fle) =f0) + xAf0) + δ arpa) - 5 Ξ )6 --3) gogo) 


- 


x(x — 1)(x —2)(x — 
+a 95 9 0 Ξ 9 avo) ....(6.2.7) 


between the appropriate limits, and evaluating the error terms in 
the manner described in section 4.8. They form one of a set of 
similar results obtainable from approximating polynomials of 
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various degrees, and are given here, in preference to other members 
of the group, because they are a compromise between accuracy and 
ease of working. In suitable cases equations 6.2.6, in conjunction 
with the Picard technique, will give initial values at (6x) = 0-2 to 
an accuracy of slightly better than one in 10°. 

An even more satisfactory technique is to make use of values of 
f(x) on either side of the origin. This is particularly useful when 
central difference methods are to be used at a later stage to carry 
on the solution. The relevant formulae can be obtained, either by 
integrating equation 6.2.7, or from Everett’s formula. For the five 
points (— 2, — 1, 0, + 1, + 2) the results are : 


[faa = GP 297(— 2) + 124(— 1) + 24/00) + 470) —F2) 


-- 2(δ1) 
41(Sx)® 
— “3600 4 m 
[ fladax = δ [—19p(—2) +846/1—1) +4960) 74/0) ἘΠ} 
—(dx) 
11 (8x) 
~ 3440 S, 
(dx) (8x) | | 
[χὰ = 555 CL—2) — 74/1) +456 /0) + 8460) — 19/02] 
+ se “fo 
26x) (Sx) 
J fled = Sp τουτ-- 2) +4/(— 1) + 24700) + 1241) +2972] 
0 
Be 


....(6.2.8) 


Although we have given the Picard method and certain integra- 
tion formulae for its application, it is not our practice to use this 
technique unless the direct analytic integration is possible. We have 
found that the Taylor series approach is more easily applicable 
and is, furthermore, readily extended to equations of higher degree 
than the first. To take only one example, consider: 


y” =f(%, 9,9’) . ++ +(6.2.9) 
with initial conditions x = xo, » = Jo, 9’ = 94. ' 
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Equation 6.2.9 gives: 


0 =F(%0, Yo, Yo) 
by direct substitution of the initial conditions. Differentiating: 
d of , ody , af dy 
3) = — } f = - - 5 ΕΡΒΈΣ. eee Ἔ- 
ΞΟ) ἐᾷ + By ἃ 
so that: 


0-2) x 


and so on. 


6.3 EQUATIONS OF THE FIRST ORDER 


The ‘classical’ method of solution of a first order differential 
equation: 


d 
a πω) τιν (63.0) 


subject to one-point boundary conditions, is that of Runge. It is 
based upon an idea originally due to Euler and has been extended 
by Kutta, Heun, and Piaggio. The method is normally used to 
obtain an initial solution with which to start one of the Bashforth 
and Adams type methods. Since, in our opinion, it has no advantages 
over the Taylor series method or the numerical version of the Picard 
process, we shall do no more than state the relevant formulae in 
the two most useful cases. 


The first Runge-Kutta approximation is the equivalent of 
Simpson’s rule applied to the integration of the f(x, y). If, in equation 
6.3.1, we know that γ = yg when x = x, and require y,, the value 
of y when x = x, + (65x). Then: 

Jr = Jo + (hy + thy + ks) -+++(6.3.2) 
where 

ky = (8x). {xo Yo) 

ky = (8x). f(x» +4(8x), γο + $4) 

ky = (8x). )ῖχο +(8%), »ο + 21, — ky) 
the error in y, being of order (5x)4, 
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If an error of order (8x)5 is required, the Kutta fourth order 
process: 
Ir =Io Ὁ oh + 21, + 21, th) ....(6.3.3) 
where: ky = (dx) f(x9, }»9) 
ky = (dx) f(xo + 4(8%), ¥9 + ἐκ) 
Ks = (0x) f(xo + 4(8x), γὸ + 44s) 
Ky = (3x) f(xo + (5%), γο +43) may be used, 


By increasing the number of steps, still higher accuracy may be at- 
tained, but the formulae are considerably more complicated and have 
not been much used. For a more extensive treatment of Runge- 
Kutta formulae the reader is referred to the standard literature. δ) 

A major criticism of the Runge-Kutta method of solution lies 
in the fact that it contains, in itself, no means of checking either 
the accuracy of the solution obtained, or the actual arithmetic by 
means of which such a solution is derived. The objection is removed 
in the Bashforth-Adams process for continuing the solution, which 
we shall now describe. 

We first notice that the method initially uses the values of the 
function previously computed to obtain new values and then uses 
the new values, and the equations, to check and correct the result. 
The differential equation itself is used as a source of values of the 
derivative. We shall require an extrapolation formula which uses 
only differences which can be computed from existing function 
values, and this implies the use of backward differences (see 
section 3.1). Now: 


fix) = E5f(0) 
and V =(E-I)E 
whence fe) =f ( — 7) 
-[1 +20 + Sys pe eS . | £0) 
(6.3.4) 


Now 
i 
| fadde = [1 Ἐὰν τὴν" +3! + VS 
Ἔν" a88279+ |. 1/10) 
or, for interval (6x): 
(82) 
| flsdde = δὴ Ὁ ἂν Ὁ ἐν" HEATH VET. . 0 
. ἘΝ ἐπ το 
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To solve the equation: 


= f(x, ») ory, = Jo +f f(x, y) dx 


we assume that five initial values, (x_ 4, y_4), (x-9,9—3)s (*-25 »..), 


(x_1, ¥-1), (Xo) %), Say, have been calculated by one τὰ the methods 
previously described. From these values the table 


ee _ ἀν ἀν. 68 ho ae ὉΠ ὐὐ ὺ ὉΣ 

e-~« J-a Fue se) aes od fe 

Fug Jus Jug Vis 

ΨΚ J—s Sus Vins Vif_s 

ey Ja f- Vf Vif 

= ho OUR OOH Vi 

“ ik | OME Ph σὰ. eH ἡ 
is constructed, as far as the broken line ---—-. Using the values of 


V'fo up to r = 4 in the integration formula (6.3.5) the value 7, may 
be obtained; this enables ἢ to be calculated and, once this is known, 
the differences \7"f, can be evaluated. 


To check the accuracy of the prediction we may form the integral: 


—(§x) 
| fle) = — (&) (1-30 - ἀν"..- ἐν: 


- ἠδ ν" — τῦνν" -- δϑέδον Ὁ" ee fC) 


or, in terms of »,, y, and the differences below the broken line which 
can now be calculated: 


n= Jo +(8x) (fA -- ἰνῇ -- ὧν ἮὮ -- AVA 
-- ἡἠδνν Ἢ -- τον Ἢ -- κἔξξυν ἢ...) .....(6.8.6) 


This formula, up to and including\/‘, is over ten times more accurate 
than equation 6.3.5, so that, if the two values of y, do not differ by 
more than 5 units in the last decimal place required, we may accept 
the value derived from equation 6.3.6 as correct. If the deviation 
is greater than this, the new value of y, can be used to recompute 
Fis VA» ete. and equation 6.3.6 re-applied until agreement between 
consecutive values is reached. 

The process is now repeated, using subscript 1 instead of 0 as 
previously, and y, is thus derived. 
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A modification of the Bashforth-Adams method, due to Milne 
will now be described. The central idea is again to predict the value 
of y,4 3 from those of y, and earlier ordinates, and then to correct it 
(if necessary) by means of a comparison formula which will, in 
general, involve y,+, itself. Milne recommends that the open 
Newton-Cotes formula, given by equation 4.3.10, be used to ‘pre- 
dict’ and Simpson’s rule (equation 4.3.4) to correct. These formulae 
become: 


‘Predictor’ 
In+1 =In—a + F(8%) (2fn —Sn-1 + 20.--) +EG(8%)8fim  «...(6.8.7) 


‘Corrector’ | 
In+1 =Jn-1 + = (fn+2 + 4fn +Sn—-1) — Je (δι). ....(6.3.8) 


in terms of the solution values y, and the values fin equation 6.3.1. 
The interval, (8x), should be so chosen that the two values of 7,1; 
differ by less than 14 in the last decimal place required. If this is 
done, a comparison of the error terms in equations 6.3.7 and 6.3.8 
shows that the value derived from 6.3.8 can be taken as correct 
within the number of places required. 

We conclude this section by describing a procedure which has 
been described by Mine) and, in modified form, by HARTREE®). 

The equation is, as usual, 6.3.1, and from it we can obtain: 


y' =f 
y= τ = -ξ ....(6.3.9) 
Now the Taylor series expansion, based on yp, gives: 
ὃ 
κει = In + (8%) In + ας “ay Ja + Said In 
ΩΝ ΓΝ + ....(6.3.10) 
whence: 
-- γδήνκμι = — HBa)yh — 4162) 9, — 9. SP ye 
a: oT MP Bk das 
: ἣ , Cr 
+ fx8x"(y)na = iz (5x) %y_ + va(8x) yn? + ve. at... 
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or, adding 


Int =In + ἐ(δα)( αι + In) — Te(8*)*( Jnaa —In) + Re 
.ο.ν.8.3.}}} 
where R, can be shown to be given by: 


R, = (8x)5 y®/720. 


Again, by forming the sum of: 
—, . (Sx)? , . (8x)3 he 
τ τὸ pee, ars: *) In +5 *) »»α (Ὁ y+... 
2! 3! 4! 
Sx)2 3 
— 2yp = — 2yn_1 — 2(5x)y,-1 — 2 se In-1 — 2 ἑοῦ In-1 
—2 Oe es -- 
, Sx)? , Sx)3 
—JIn-2 = — »κ--ὶ + (8%) ,-1 — St wens + ( =! In-1 
Sx)4 
sei Ina ᾿ ΝΗ 
2 3 
— 4(82)5% = τς. — Ss 
(δὴ. 
4 n—1 


and then substituting for the derivatives γη, 9%, y{?, in terms of the 
Taylor series based on ,_1, it is easily seen that: 


In+1 =In-a + 3( In —In—-1) + (8%) " rn — In—1) + Rp 
..«.«ν(6.,3.12} 
where 


Ry = 60(8x)5y/720. 


Formula 6.3.12 can be used to predict the value of y,., given those 
Of γη5.» Jn—1 and yn, the differential equation itself being used to 
determine γᾷ and γῇ... (from equation 6.3.9). 

From these predicted values a corrected value can be calculated 
via equation 6.3.11. If the difference between the predicted and 
corrected values is less than 30 in the last decimal place required, 
the latter is taken to be correct. If the difference exceeds this figure 
the corrected value is used to recalculate the derivatives y/,,, 
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Yn+1 in equation 6.3.11, and a new and more accurate value of 
Yn+1 18 thus obtained. This process is continued until two successive 
values of y,+, do not differ within the limits of accuracy required. 
It should be noted that, with a properly chosen value of (5x), a 
re-application of equation 6.3.11 should not often be required. If 
it is, there is a clear indication that a change in (6x) is needed. In 
a similar manner, if repeated use of 6.3.11 is never required, it 
suggests that too small a value of (6x) is being used. A working rule 
is that one re-application of equation 6.3.11 should be required 
every five or so steps. 


6.4 EQUATIONS OF THE SECOND ORDER 


The canonical form of the differential equation of the second order 
is: 


d? | 
7 = fix, 9) 2. (6.4.1) 


and any equation of the second order, linear in the derivatives, can 
be reduced to the form of equation 6.4.1. Thus: 


d%y d 
sa + a(x) 5 + B(x, 9) = 0 


reduces to 6.4,1 when the transformation: 


y = Y exp [— fl a(x)dx] » 00 (On) 
is applied. The transformed equation is: 


Y” = [ha’(x) + 2{a(x)}*] Y — d(x, Y)exp [a(x)dx ....(6.4.3} 


but unless b(x, }) is of the form _y.c(x), this is not likely to be suitable 
for numerical calculation because of the exponential factor. Further- 
more, the transformation destroys any periodicity which may exist in 
solutions of the original equation. ΓῚ 

By means of the transformation: 


z= γ' ..- (6.4.4) 
the general second order equation: 
) =f 7,7’) εν. +(6.4.5) 


is reduced to the pair of simultaneous first order equations: 


zg’ = f(x, 9, 2) } ...- (6.4.6) 


f= γ' 
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which may be solved by the methods discussed in section 6.5. This 
reduction has the merit of simplicity, but is not to be recommended 
for equations in which γ' is absent. 

The Bashforth-Adams method can be applied directly to equation 
6.4.5 when a suitable set of initial values has been obtained; the 
detailed procedure is as follows: 

(1) Assume that γῇ, yn; - . . etc. are known. 

(2) Find »;,,, from γᾷ, γᾷ... etc. by means of 6.3.5. 

(3) Assume that y,,, y,_) etc. are known. 

(4) Find »,,) from_y,.1, 9%, . . . etc. by means of 6.3.6. 

(5) Using these values of 7,41, ¥,41 compute y,.; from 6.4.5. 

(6) Recompute y;,.1 ¥,_4.1 by means of 6.3.6. 


At this stage the usual error estimation can be made and, if neces- 
sary, one or more extra cycles of the process can be used to obtain 
the desired accuracy. 

The Bashforth-Adams backward difference formulae are not the 
only ones which can be used in this process. Thus Mine) suggests 
that the open Newton—Cotes formula (4.3.10), in the form: 


Int = In—s + F(8x) (2m — Yn—1 + 2n—s) 
should be used in step (2) (supra), and that this should be followed 
by Simpson’s rule: 
(5x) fos ’ , 
Invi =Ja-1 T Ἔ (In+1 + 49n + Yn-1) 
in step (4) and again in step (6), this time as: 


f r ox i i ul 
In+1 =In-1 1 τὸ (Pn+1 + 49, + Jn—-1)- 


For second order equations in which the first derivative is absent 
(i.e, the canonical form 6.4.1), a simplified procedure is available 
which, in effect, uses a double integration formula to pass directly 
from 5" to γ. 

One such formula has been obtained in equation 4.5.3; it may be 
rewritten, for the present purpose: 


κει = 276 —JIn-1 — (x)? (In + 729*y;, >. 7200 Yn + es .) 


ves AGT) 
or 
ἃς (δι), , a shite - 
Inti = ὅν, — In-1 + 9 (»»ιι + 10%_ Ἔ}...ι) — τισί δι) yn. 
....(6.4.7α) 
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This result may be used as a corrector and the less accurate: 
In+1 =In TIn—2 —In-8 
1 OO axe + Qh + Soha) + ἐδίδουν, (6.4.8) 
as a predictor. As an alternative to equation 6.4.8: 


Int1 = Wa—-1 —In—3 + 4(5x)? (Hn—-a + ξδ3γῃ...) + AS (8x) OR 
..«..(6.4.9} 


which may be regarded as the analogue of equation 6.4.7, but for 
double the interval, is perhaps simpler to apply. 

Another, and more recent, method for solving the canonical form 
of the second-order equation is that of DE VOGELAERE"’, 

The equations used are: 


Yny = Yn + 4(8x) yn + ga(5x)*(4yn — yng) + O(8x4) 

Yn+1 = Yn + (δα νη + 3(8x)?(Yn + 2ynay) + 0(8z5) 

Yuet = Yn + ἐ(δχ) (Yn + ney + Yn4r) + O(8x5) 
which, as in the Runge—Kutta process, allow the progressive solution 
of the equation. The advantage of de Vogelaere’s method is that it 
requires the evaluation οὔτ = /(xn,y,)) only twice per step. 


To start the solution it is necessary to know ψ0 and yj and y”_,. It is 
usual to compute the latter from: 


ψ- = Yo — $(dx)yo + §(8x) yo. 


When the form of f(x, ») in equation 6.4.1 is such that its calculation 
for any particular values of x and » is difficult or tedious, a procedure 
due to Jennings, Fox and Goodwin often enables a large interval by 
differencing to be used, with consequent reduction in the number 
of values of f(x, ») to be calculated. In essence, the method uses a 
large value of (5x) and a simple formula to obtain a rough solution, 
and then uses the differences obtained from this solution to enable 
a better approximation to be made. 

Thus, suppose that a set of values yj, ,) at interval (8x) have been 
obtained; then, from equation 4.5.3 


δὲν = (5x)? (γ΄ + ὑχδῆν" — gap8" + - - +) 
and, if we estimate δ3γ", 54y" from the previous approximation, 
og "Ὁ (dx)? [my τ ὦ 758 *¥m—1) = εἷς 8Sin—1) : .} 
or 8°ym) — (dx)? [705 Ym} + 720 °¥(m—1) — 5400 ¥m—1) εν. 
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Thus, if (m)n+1> νην Jonn—1 are consecutive points in the mth 
approximation: 


Hmyn+-1= 2% myn — Penyn—1 


+ (dx)? LA; Ymyn} 5 259 7V(m—1)n = 5400 Y(m—1)n κα αἱ 
.....(6.4.10) 


a formula which enables the solution to be continued from the nth 
point of the mth approximation to the (n + 1)th point. 

The same general method can be applied to equations of the first 
order and a comparison of such techniques has been made by Fox 
and Goopwin"!)’, A particular calculation, in which an economy of 
labour has been achieved, is the tracing of paraxial rays through 
an electron lens system where the field distribution is given by 
complete elliptic integrals which are awkward to manipulate. This 
problem has been extensively studied by Jenninos"!?? 


6.5 SIMULTANEOUS DIFFERENTIAL EQUATIONS 


The typical set of simultaneous differential equations of the first 
order may be written: 


Ke lle, He Ness eB), (eels so 5). .-...(6.5,}} 


The methods of solution are almost identical with those suggested 
for the ordinary first order equations, and are typified by the 
following scheme. 


(1) Obtain initial values from a Taylor series expansion and the 
given boundary conditions. 

(2) From yp, ο, Imo (m=1... mn) predict values of ym, by 
means of equation 6.3.5 or 6.3.7. 


(3) Using the values of ym, (m= 1...) obtained in (2) re- 
calculate y,,, from the differential equations and 6.3.6 or 
6.3.8. 


(4) If significant changes occur repeat the process from (3) until 
adequate accuracy has been obtained. 


It should be noticed that this technique can be applied, as indicated 
in section 6.4, to obtain the solution of a second order equation. 
In a similar manner, the set of equations of the second order: 


Ym = fin(*, Yo D2 eli In) (m ΞΕ}, ,. n) ++ ++ (6.5.2) 


may be solved, using the repeated integration formulae of equations 
6.4.8 and 6.4.9 as predictors, and 6.4.7 as a corrector. 


72 


MULTI-POINT BOUNDARY CONDITIONS 


6.6 EQUATIONS OF HIGHER ORDER | 

When such equations, either single or simultaneous, are encountered, 
the only really practical method of solution appears to be to reduce 
them to sets of first order simultaneous equations by means of suc- 
cessive transformations of the type in equation 6.4.4. The actual 
integrations at any step should proceed from the highest order 
derivative downwards and (6.3.5, 6.3.6) (6.3.7, 6.3.8) or (6.3.11, 
6.3.12) are suitable pairs of integration formulae. 


6.7 MULTI-POINT BOUNDARY CONDITIONS 


Although we shall defer consideration of this type of solution until 
Chapter 8, section 6, it is appropriate to indicate here the mode of 
application of the methods just discussed to this problem. 

First consider a second order differential equation 


9" = f(y 9, x). ....(6.7.1} 
If this is to satisfy: 


JY = Jo at χ = Xp 
yHpatx=xy 


there is, of course, insufficient data with which to start a finite differ- 
ence integration from x = x9. Suppose, however, that 6.7.1 is a linear 
equation; then if y =/,(x) and y =/,(x) are any two solutions, 
y = Al,(x) + Bl,(x) is also a solution. 

To solve the given equation all that is needed is the following. 
From x = x, start any two integrations having y = 9 but different 
values of γ' at x = xg. Carry these solutions up to x = x, and assume 
these solutions to be /,(x) and /,(x). The required solution is then: 


y = Al,(x) + Bl,(x) 
where A and B are the solutions of: 


Io = Al,(xo) + Bla(xp) (i.e. A + B= 1) 
Jy = Alj(*) + Blam). 

When equation 6.7.1 is not linear this technique will not work, 
since Al,(x) + Bl,(x) is no longer, in general, a solution of the equa- 
tion. In this event, a possible method of approach is to guess a pair 
of initial directions which produce solutions straddling the required 
values at x = x,. Linear interpolation then gives a better initial 
direction, and the process is repeated until the desired accuracy is 
attained. Since most of the non-linear equations encountered in 
practice, have solutions which vary exponentially as some power 
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of the independent variable, this method is, however. seldom 
applicable over any considerable range. 

An extension of this procedure is to take more than two trial solu- 
tions and then to use a non-linear interpolation procedure to obtain 
the correct initial value. 


6.8. EIGENVALUE PROBLEMS 


It is often required to obtain the ‘eigenvalues’ of a differential 
operator. By this is meant those values of λ, for which the equation: 


Lafin(X) = λιψ, (α) ....(6.8,1) 
has solutions which satisfy specified conditions at x = a and x = 6. 
A typical form“ for the operator L is: 
d2 
sa a ΣΕ .. ++ (6.8.2) 
and (a, δ) is often (0, 00). The trivial solution (x) = 0 is excluded, 

Depending upon the form of L, and of the boundary conditions, 
A, may have discrete values only, or a complete spectrum. The de- 
termination of the values of A, can be carried out by the process of 
constructing trial solutions for different assumed values of A and then 
following the process outlined in section 6.7 to obtain more accurate 
values. 

It should be emphasized that numerical calculation of eigenvalues 
should never be used as an alternative to a proper analytical investi- 
gation of the equation. In many real physical situations such an 
investigation will lead to a fruitful result. 

An alternative approach to the determination of approximate 
eigenvalues is via the equivalent set of simultaneous difference 
equations, but this is dealt with more fully in Chapter 8. 
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7 
SIMULTANEOUS LINEAR EQUATIONS 


7.1 PRELIMINARY REMARKS 


ALTHOUGH simultaneous linear equations arise in a straightforward 
manner in many problems of physical and engineering science, they 
have attained a much greater importance in recent years because 
of their application to the solution of various sorts of differential 
equation—both ordinary and partial. 

Whereas the ‘classical’ application of simultaneous equations, to 
such problems as an ordnance survey, rarely results in a set of 
more than 10 equations in 10 unknowns, many of the more modern 
approaches to differential equation theory deal with 100 x 100 sets 
as a commonplace and 10° x 10° and upwards as a desideratum. 

Classical methods of solution by elimination or determinants 
have, in some hand computing at least, given way to approximate 
iterative techniques. 

The subject, although of great antiquity, is still giving rise to a 
spate of research papers and there appears little agreement as to a 
best method of approach. The reader who wishes to pursue the 
subject can refer to a recent survey and bibliography ® in which over 
450 references to the literature are given. 


7.2 DEFINITIONS 
The set of simultaneous equations: 


(1) @yy% + Qyo%a + Qyg%g . νον νον AnX, = by 
(2) ἀργλ + ἀγ)χᾳ + Gggtg. ss. ἄφηχῃ = by 
= AgyX% + . 
ehmisioelel ms ἃ καὶ ww a Φ κα α τὰ “4 {27.8.8} 
γ). AryXy + ArgXq + + AppXp Ary Xp_ = dy 
(a) Guy + Aneta tess ee ee Ἔ GunX,= 5, 
is taken as the basis. It may be written, in matrix form: 
A.x=b samathamee) 
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where: 
Gy, -Ayg + + . ἄγῃ xy 
Gg) - Ag - Gen Xs 
A=m|*m* ++ °s Gan) χ =| 73! ὃ ΠΤ TQS) 
Gny-Ong + + Ann Xn 


Associated with A, are the determinant |A|, the transpose A’ [if A = 
(ay) then A’ = (a;;)] and the inverse A—* such that: 


A1,A=A,A1 =] εν. (7.2.4) 


where J is the unit matrix: 


i ooo ὃ 
0100 0 

Pel fiche 1. ὃ o|. ΤΥ 
0000 1 


Formally, we have from equations 7.2.2 and 7.2.4: 


A-1.A.x = Ab 
but 

A. Ax=I.x=X 
whence 


x = 4 18. ... (7.2.6) 


It is shown, in works on matrix theory), that: 


Ay/|A|, An/|A] « - - An/|Al 
4.4], AnilAl - Bioniit 


..(7.2.7) 


5 Ίκϑκ5 5 δα 55 & F&F ἃ ἃ © ££ ἃ δ ἃ HE ἃ 


Aj,/|A|, Ag,/|A| ie Ann/|A| , 


where A,; is the co-factor of a, in | A]. 

In the above, and in the following sections where the condition 
is necessary, it is assumed that |A| 4 0. When this condition is 
satisfied, the matrix A is said to be non-singular. 

When ὃ = 0 (i.e. b,, (7 = 1 . . . πὴ = 0) the equations 7.2.1 are 
said to be homogeneous; in this case the necessary and sufficient con- 
dition for non-zero solutions is |A] = 0. 
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Associated with A is the characteristic equation: 
A.X=A.X 
or: 
(A —Al)x =0 oe eel tee) 
where the A’s are scalar quantities. 
Solutions of 


a,—A Ay a3 ἄγῃ 
|A—Al| = aq) Gog — A ag Aon —0 
| Qny δον τυ ee es Ann — A 


are known as the /atent roots of A (sometimes called eigenwerte, eigen- 
values, or characteristic numbers). Clearly both A and A’ have the same 
latent roots. 


x and 6 in equations 7.2.2, 7.2.3 are seen to be composed of n com- 
ponents, x, . . . Κρ; 6, . . . 6,, and for this reason are often called 
vectors. Associated with each latent root, d;, say, is a latent vector 2; 
which satisfies: 


(A — Ad )p, = 0 (ἐ τα 1... ni). 
It is easily verified that (Ax)’ = x'A‘ whence, if: 
Ap; = λιφι 
(Acp,)’ = pA’ = Axo; enon lame) 


Now let the latent vectors of the transpose of A, A’, be called ψ, then, 
by definition: 


A's; = Aph; (j= 1 ss . n) 
whence, pre-multiplying by ¢; 
PA's = λιφίψι, 
similarly, post-multiplying 7.2.9 by ;, we have: 
PA; = λιφίψ; 
or, subtracting the last two relationships, 
0 = (Aj — Ai) φίψ;- 


It follows that, if Aj 4 Ay, p;. ys; = 0. The vectors φινψε are referred to 
as hiorthogonal. 
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When A is symmetric, so that A’ = A, it is evident that @; = yf; 
(i=1... n). In this case φί. φ; = 0 (i τέ 7) and the latent vectors 
are orthogonal. It should be noticed that each of the latent vectors is 
arbitrary to the extent of a scalar multiplying constant. If these 
constants are so chosen that τὶ φι = | (i= 1... 2), the latent 
vectors are said to be normalized and constitute an ortho-normal set. 


Finally, with each matrix A = (a;;) is associated a quadratic form 


nm Tf 
2 pd Dig XiX; (ay; = aj) o nse εῦθι 
which is said to be positive definite if it is positive for every set of real 
(x;) except (x,=%,... =x, =0). (Negative definiteness is 
defined in a similar manner). 
A set of necessary and sufficient conditions for positive-definiteness 
is: 


a) Oe ππὸ 
Ao, 499 | 


a, > 0, Me SORE i> ὃ, ΓΞ) 


7.3 EXACT SOLUTION 
The exact solution of the equations 7.2.1 is easily written down; 
it is: 
32 + Mr—yO yO rt1 + Sn 
| em " Gor—yDo4or+1 . ἄρῃ 


s « ὃ ἃ ἃ ἃ ἃ ἃ ἃ ὃ £&+ ἃ ἃ δ Ff 


| ἀχγνᾶχα + + ἄχη 
ΑΜ δ8ὺ 


- ὃ ἃ ἃ ΓῚ 


ἄρχυῃᾳ 2% ann 

Although this is formally the complete answer, it is completely use- 
less for practical computation. The reason for this lies in the fact 
that, to evaluate a determinant of order n from its algebraic defini- 
tion requires (n!)(n — 1) multiplications; it follows that the evalua- 
tion of all x, in equation 7.3.1 would need (n + 1).(n!)(n — 1) 
multiplications and n divisions. Other methods of evaluating the 
determinants exist, for example that of Doolittle, but these require 
roughly n3/3 multiplications per determinant, so that the whole 
solution still requires about n‘/3 multiplications. (Notice that in all 
of these estimates we assume that division and multiplication are 
roughly equivalent in computational labour, and that n is large 
compared with unity.) 


A more economical method is the following: consider the Ist and 
rth members of the set of equations 7.2.1. Multiply the former by 
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a,, and the latter by ἀμ), and then subtract (1) Χ a, from (r) X a. 
The result is: 


(α,γάγᾳ — yyy) %q + (ἀγγά,ᾳ — G71 13)Xq «αν 


Ἔ (ym — ἀγχαγ,) Xn = (αχχιδ, — ayyb,) 
or 


trata + alrg¥g + + + + + alnt = 1b, Say 


Repeating this operation for all r in the range (1 <<r<n), a set of 
(n — 1) equations in the (n — 1) unknowns x, . . . X,, areobtained 
and a total of 2n(n — 1) multiplications is required. (We do not need 
to form @,,4,, since this coefficient is eliminated). This sequence of 
operations is now repeated successively until x, . . . %,—-, have each 
been eliminated; the result is a series of equations: 


(1) Gyyr%y + ἀχοΐς + ἀγοΐα - - 1 ee + OynXn = δι 

(2) tloe%e + alestg- + + + +s tlontn = 152 

(3) alsa%3 + alaaXq- - siti = ob, (7.3.2) 
(πὴ (n—1)!nn*n = (n—1) bn 


The total number of multiplications involved, up to this point, is 
2E(n)(n — 1) + (n —1)(n— 3) « «« 2.1] = ξπ(πϑ — 1) 


Starting with the nth member of equation 7.3.2 we can find x,, 
Xn—y) X,—g «ον X, successively by substitution of the values of x, 
already found in the next equation [i.e. x, in (n — 1) to get xp_,; 
Xny Xn—1 in (n — 2) to get x,_, efc.]. It will be seen that this process 
involves 4n(n — 1) multiplications and n divisions, whence the total 
number of ‘equivalent multiplications’ for the whole solution is: 


n(n? — 1) + Ann — 1) Ἐπ 


which is of the order $n° for large values of n. 

The merit of this process lies in the fact that, if the initial coeffi- 
cients ἀρ are small whole numbers, no round-off may be necessary 
during the initial elimination; thus if integer solutions exist they will 
be obtained correctly. (This may be important if the equations are 
‘ill conditioned,’ vide infra.) 


When the initial coefficients are not integers a more efficient 
process is as follows, 
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(1) Examine the matrix A = [aj] and find the largest aj, call 
this ayy. 

(2) Multiply the Jth equation successively by all a,;/azz (¢ = 1 
. . . n #TI) and subtract from each of the other equations. 


The result of this operation will be a set of (n — 1) equations from 
which x, has been eliminated, just as occurred in the previous 
method. The same discrimination—elimination process is applied to 
the new set of equations and continued until a triangular set of 
equations of the same type as 7.3.2 is obtained. Back substitution 
then yields the x,. 

The reason for the choice of the largest a;; for the divisor at each 
stage is that the round-off errors, generated at each formation of 
a;;/azz, are thereby reduced in size at the next stage. The method 
is sometimes described as ‘pivotal condensation,’ the element a;,; 
being the pivot. 

It is easy to see that n(n — 1) multiplications and (2 — 1) divi- 
sions have to be made at the first elimination. This gives (n* — 1) 
‘equivalent multiplications,’ compared to 2n(n — 1) in the previous 
process. Thus, for the whole solution, a number of multiplications 
of order $n* is required. . 

For hand computation, pivotal condensation is an excellent 
method since the largest coefficient a;y; can usually be seen at a 
glance. When an automatic computing machine is in use, however, 
this discrimination may be troublesome and, for this reason, it has 
been our practice to use the former method whenever possible. 

A third method of solution is usually known as the Choleski 
process; it depends upon the reduction of the matrix A to the 
product of a lower triangular matrix Z, with unit diagonal co- 
efficients, and an upper triangular matrix U. 


Thus: Ax=L.U.x=6b 
Ux =€& 


and the solution is reduced to a pair of back substitutions. The co- 
efficients /,; and u,; are readily obtained by forming the product 
L.U and equating coefficients to those of A in the sequence 

(ἀν τος Oy lag: . «δῶν ον (ἄχ ess gs ον (δεν ee 


It is stated that an advantage of the Choleski method lies in the 
fact that few intermediate results have to be written down; it is our 
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experience, however, that the method is too complicated for 
‘occasional’ use, and is not suitable for use on an automatic 
computer. 

7.4 THE INVERSION OF MATRICES 


When it is desired to solve the equations 7.2.1 for a fixed set of aj, 
but for a number of vectors 6, it is best to evaluate the inverse 
matrix A~? and to use equation 7.2.6 to obtain the individual 
solutions. 

The formal inverse (7.2.7) is useless, for the same reasons which 
cause the rejection of the determinant solution of the original 
equations. The elimination method may, however, be extended to 
give the inverse of a matrix. Thus let: 


QqyyyFyg - + . Ayn 
Aq} = | [22+ + + Fan » wash tet) 


= = ἃ ἃ + ἃ «ff 


Then 
* ἀμ» - + « Gn [] δι yy), + αγρὺς + αγεῦς . ..ὄ + Ayndn | 
oe: ἢ @ai,0g9- + - Gon || ὅς Gaby + Agabg + ἀγεῦς... + ἀφεδῃ 
Xn GnpOne: - x: ἘΞ by An), + Anos + Angls os + ἀρμηδῃ 
...«.(7.4.2} 
Whence, if we solve the original set of equations for the vectors: 
[1] [0] fo] fo 
0] l 0 0 
0 0 l 0 
ὃ =/0 ΠΥ - ave CES 
0 0 0 l 


the solutions, considered as column vectors, are the columns of the 
inverse matrix. 

We note that by using pivotal condensation to solve the sets of 
equations and arranging the work so that only one triangulation of 
Ais performed $n’ multiplications are required. 

The Choleski method may also be applied to matrix inversion, 
for let A-m LU 
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where 1, and U are triangular matrices of the type defined in 
section 7.3. Then: 

AA =] 
whence: L.G.A+=TJI 

L3ALUA— = UA = L-YU =—L-! 

U-"U A = At = UL, 
Now it has been seen in section 7.3 that U and L can be determined 
from A; assume that: 


Then, since L.L~! = J, we have, equating coefficients: 
λῃ = 1, Ay =Ayg =- + - An = 0 
Ayer + Agr = 0; — Agoler + Ase = 1, 


which will enable the A;; to be determined. In a like manner the 
components γὼ) of the inverse {71 may be determined. Thus, multi- 
plying U-12-, the inverse A! is obtained. 

Two other methods of finding an inverse are worthy of mention, 
not so much for their efficiency as far as numbers of multiplications 
are concerned, but because they are very well adapted to punched 
card and automatic computing machinery. 


The first method is a consequence of the Cayley-Hamilton 
theorem (‘A matrix satisfies its characteristic equation’). 


Let N+ aA" + a2... tay = 0... (7.4.4) 


be the characteristic equation of a matrix A. Then: 
A" + a,A"-! + a,A®-?+...+ 4, =0 
whence : T= — © (dy + aA"! + agA™-2, . «+ yard) 
and: 
A-Yf = A = — = (A"-1! + a,A"-2? + a,A"-3 +... + ay_y/). 
....(7.4.5) 
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Now it may be shown that: 


ἅη ΞΞ -- δ. 
a, = — Has, + 59) 
a, = — 4 (ays, + 4,5, + 55) .. (7.4.6) 
ἄμ = (a, —151 + ἀμ... Se +n) 
where: 
5, = tr(A") .... (7.4.7) 


n 
ir being the frace of the matrix (i.e. 2 a, the sum of the elements 
i=1 


on the principal diagonal). 

Since a single matrix multiplication requires n* scalar multipli- 
cations the process of inversion embodied in equations 7.4.5, 7.4.6 
and 7.4.7 will require of the order of πα multiplications. 


The second inversion technique is an iterative one based upon 
the well-known result: 


Χμ = Xn(2 — ax,) ..++(7.4.8) 
for which 
Lt x, = l/a 
NFO 


Under suitable conditions) it may be shown that a similar 
iteration: scot 
An+1 ΞΕΞ Χ, (Δ] παν A.Xn) eeee (7.4.9) 


in which X, and A are matrices and J is the unit matrix, will con- 
verge to A~!. A practical experiment (4), in which this method was 
applied to the inversion of a 16 x 16 matrix, showed that 10 itera- 
tions were required to produce 4 decimal place accuracy and 20 
iterations to produce 6 decimal place accuracy. Actually, the rate 
of convergence depends upon the goodness of the original approxi- 
mation, X5, and upon the precise form of A. It has been suggested 
that X, = J should be taken as a starting value. 


7.5 RESIDUALS AND ‘CONDITION’ 


With the exception of the last method described, the procedures so 
far outlined for solving a set of linear simultaneous equations will 
give an exact solution if the arithmetic operations can be performed 
with complete accuracy. In practice only an approximation will, in 
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general, be produced, since the arithmetic operations x and ~ 
will be subject to round-off errors. The iterative methods which will 
now be described make no pretence at complete accuracy, but 
merely reduce the errors in an existing approximation. This leads 
naturally to a consideration of the way in which the accuracy of 
a given approximation may be measured. 

Let us suppose that an approximation —&(= é,,f., .. . &,) is 
given to the solution of the set of equations 7.2.1. If we substitute 
this in the given equations we obtain: 


yf) + bet. -  ἀγιξη — b= 7 


Aasey + ἀρεῖς +. . . + Gon’n — ὃς = Te .(7.5.1) 


ἄρχξι + ἀμᾳξς +. . . + GanEn — bn = Τρ 


The quantities r; define a vector r (== 1, To, . - . T) which is a 
measure of the inaccuracy of the approximation. To obtain a single 
number which will express the inaccuracy we may consider the 
length of r, or more usefully its square, R*. This is defined as the 
scalar product of r and itself: 


R? =(r.r) = 2 = (AE —b).(AE—b) ....(7.5.2) 


and the quantities 7; are often called ‘residuals.’ 


By means of equation 7.5.1 we may reduce 7.5.2 to a quadratic 
form in the €;: 


R= Σ Pe ee eee ee ee Se ΠΝ 


which is clearly positive-definite. 

An alternative measure of accuracy is the quadratic (previously 
given in 7.2.10): 

5 τι αὶ ΣΣ Σ ayes -- 24, : (ἀρ = ἀμ) ....(7.5.4) 
i=1 j=1 

This has the merit of bearing a simple relationship to the original 
matrix A, but the disadvantage that A must be positive definite, 
so that arguments based upon S may fail if this condition is not 
satisfied. We note, here, that equation 7.5.4 may be written, 
vectorially:* 


* Notice that in equations 7. 5. 2, 7.5.5, A has the character of a ‘Tensor,’ or in 
more modern usage, a 6. 
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S = ἐξ. Αξ --- δ. τὴν ὅν 5.1 
= ἐξ. (ν - δὴ -- δ. ξ 
= 4&.(r -- δ) 
= 4(r --- ὃ). 4-τ(ν + B) < 203 7..}} 


It is clear from equation 7.5.4 that S has a minimum when r = 0, 
and the same is true of R*. 

The question now arises, how far may an approximate solution § 
deviate from the true solution x for a particular value of R? or S. 
A measure of this deviation is the square of the length of the dif- 
ference vector (§ — x), that is 


(E — x). (6 — x) = (A). (42). 
This is not easily correlated with either R? or S but it shows that if 
A-' has any large component, large differences between true and 
approximate solutions may be accompanied by small residuals r; 
and values of R? and S. 

Another approach is to consider the hyper-ellipsoids defined by 
equations 7.5.3 and 7.5.4, for fixed values of ΚΞ and S. Consicer- 
ing the latter (S), we first transform to axes through the centre 
(45, Hes s + Χρ). Ῥαϊηρ 


é, =%*%+ 6 
equation 7.5.4 becomes: 
Stik tant Z E agen. .... 05.5.2) 
inl Gul ἕω 


Now it is well known ©) that the quadratic form: 

Σάαμειε 

i Ἵ 
can, by means of a real orthogonal transformation of unit modulus, 
be reduced to: 

n 

Σ AER ....{7.5.8) 


i=l 

where the A; are the roots of the characteristic equation 7.2.9. 
Now 

π fi 

S+4 2 bx%;= 2 AEF} 

i=] i=] 

represents a hyper-ellipsoid having axes proportional to 1/4+/A; 
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so that, if any of the A; are very small, large values of Εἰ can be 


2 Minor axis = 0-1 


1,1) 


Figure 7.6.1 


accompanied by small residuals. This is shown, for a two-dimen- 
sional case, in Figure 7.5.1. 
The equations considered are: 
10x, + 10lx, = 111... (b) 
for which the solution is obviously x, = x, = 1; 

Figure 7.5.1 shows the lie of the curve for § = 60-5, the actual 
curve being too close to its major axis to be visible on this scale. 
It will be seen that the axes are approximately τὰς and 10 in accord 
with the ratioV/),/A, of the roots of: 


1—A 10 


10 101 —A|\=° 
or 
A? — 102. +1 =0 
giving 


A, = 101-9902, A, = 0-0098. 


Also plotted are the lines represented by the equations (a) and (b), 
and it will be noticed that they are nearly parallel, so that the 
intersection (1, 1) is very sensitive to the coefficients. The relation- 
ship between residuals and parallelism is the basis of the definition 
of R? in equation 7.5.2; thus if p; is the length of the perpendicular 
from (&, € . . . &,) on to the hyperplane: 


AiyX, + GigXs + Gigts « « - GinX, — δι = Ὁ 
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we have: p= (2 ati, — ws) /4/ Σ ai; evant tne) 
= | j=l 


—— 
= ff 2 ay. 
τι V 5 a 


If now we normalize the rows of the matrix A 


(is. Sve cach: row by|/ Σ a) 


we shall not alter the solution of the equations. Assume that this 
has been done, so that equation 7.5.9 becomes: 


Di = T% ....(7.5.10) 
Thus equation 7.5.2 may be written: 


Tt 
R? = 32 pi? 
ἐπὶ 
and we see that the surfaces of R? = const. are simply those such that 
the sums of the squares of the perpendiculars from any point 
(ὦ, 9 . - - &,) on the surface on to the hyperplanes 


QiyX, + Gigkg «0. Ginkn—F Ξε Ο (ἐπε 1... 2) 
is constant. 

It will be evident, from Figure 7.5.1, that in the example quoted, 
the R? = const. ‘surfaces’ are similar in shape to those defined by 
S = const. 

The equations (a) and (b) provide an interesting study of the false 
conclusions which may be drawn from small values of R*. We first 


normalize by forming (a) //1* + 102, (b)//10® + 101? and thus 
obtain: 


Xy 10x, ᾧ 1] , 
Viol ΤΥ Vor (a) 
10x, 10x, 111 


Viosol + Viosol ~ νΊΟϑΟΙ 5 '° 


‘The radicals have been purposely retained thus far. If we calculate 
the components of the residuals for the approximate solution 
x, = 1-001, x, = 1-01 we find: from (a’) r,=-01, from (b’) 
r, <= -01 whence R* = -0002. 

Next try the values x, = 11-1, x, = 0, we obtain: r, = -01,7,=0 
whence, this time R= -0001. Most people would conclude from this 
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that (11-1, 0) was ‘nearer’ to the true solution than (1-001, 1-01) 
which is manifestly false. 

If we had evaluated the radicals and expressed the resulting 
coefficients in decimal form, the equations (a’) and (b’) would have 
appeared as: 


-09950x, -+ -99504x, = 1-09454 (45) 
-09853x, + -99513x, = 1-09366 (b”) 


and a change of approximately 1 per cent in the coefficient of x, in 
(a”) makes the solution (11-1, 0) exact. 

Thus, the solution of equations of this type is very sensitive to the 
values of the coefficients, and the term ‘ill-conditioned’ is applied 
to them. 

Various measures for ‘ill-condition’ suggest themselves from 
the preceding discussion. Perhaps the most obvious is the ratio 
Amax/Amins Which gives a measure of the ratio of greatest to least 
axes of the hyper-ellipsoid defined by equation 7.5.4. Unfortunately 
the calculation of Ama, and Amin is an operation of at least the com- 
plexity of the solution of the original equations, so that this criterion 
is of little practical use. ΠῚ condition is accompanied by approximate 
parallelism of some of the hyperplanes defined by the equations. 
This can be recognized in simple cases, such as that obtaining in our 
equations (a) and (b), by observing the ratios of corresponding 
coefficients, but this, too, is not generally possible. 

Yet another measure which has been suggested is the value of the 
determinant | A], ‘small’ values being accompanied by ill-condi- 
tion. In this form the test is without value, since the set of equations 
can be multiplied by any constant without altering the solution. 
If the constant is M, | A] is multiplied by M" and so can be made as 
large as desired; we note that if [4] =0 the equations have no 
unique solution. A better version of this test is first to ‘normalize’ the 


equations by division by A/ Σ ai; for the ith equation, and to 
1 


j= 
regard smallness compared to + 1 as an indication of ill-condition. 
On this basis the ‘ideal’ equations: 


Qy,%, = δ, 
αφᾳῖς = by 
and αχγυχχ + ἄγ»χῳ = by 
AgiX, — ἄφοχᾳ = by 
with 14, = Ag, Gq, = 222 
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have [4] = 1 [41 = —1 respectively, when normalized. On the 
other hand, our equations (a) (b), in the normal form (a”) (b"), 
have [41] =~ -001 which gives a clear indication of their ill-condition. 
Yet another example is provided by the much quoted ©) set of ill- 
conditioned equations: 


SX, + 7x, + 6x, + 5x, = 23 
7x, + 10x, + 8x,-+ 7x, = 32 
6xj + 8x,+ 10x,+ 9x, = 33 
5x, + 7x,-+ 9x, + 10x, = 31 


whose true solution is evidently (1, 1, 1, 1), but for which the values 
(+ 14-6, —7-2, —2-5, +-3-1) give components {- -1, — -1, — <1 
+ +l) for the residue vector r, and consequently R? = -04. The 
values (- 2-36, + 0-18, + 0-65, + 1-21) give residue components 
(+ °01, — ΟἹ, — ΟἹ, + -01) and consequently R* = -0004. The 
value of |A| for this set of equations as it stands is 1, but if we 
normalize by division of the respective equations by 


ν 5: 1 75 POLS, VEPIOP RT, 

V6? + 824 1027492 and ν58- 75 + 92 + 102, 
we get |Alnorm, = *000,019,9 which, since it is small compared 
with unity, indicates clearly the ill condition of the equations. 


7.6 DETERMINATION OF LATENT ROOTS AND OF 
CHARACTERISTIC VECTORS 


It is possible to determine the latent roots of a matrix directly from 
the definition 7.2.8, and then to solve the resulting sets of simulta- 
neous equations (one set for each A;) to determine the characteristic 
vectors. This method, whilst sometimes appropriate for small 
numbers of equations and unknowns, say less than 5, is not at all 
feasible when large numbers of variables are involved. This point 
is therefore an appropriate one at which to consider methods of 
iteration and successive approximation which will, at the same time, 
be useful for the solution of simultaneous equations themselves. 

When no information is available regarding either latent roots or 
characteristic vectors, it is usual to start by determining the largest 
A; and its associated vector. 

We shall adopt the usual procedure and denote the latent roots by 
Ay, Ag. + + An, where |A,| > |Aq| > [Ag] . . . > An], the associated 
characteristic vectors are ~,, 4... . @, and we recall the definition: 


(A -- λίγφι =0 (i=1... 2) οὐ 
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Consider the effect of applying the tensor A to any vector x, that 
is, of forming Ax. The tensor operator can be considered as a set of 
expansions, or contractions, in an n dimensional hyperspace. These 
expansions have directions given by the principal axes of the quadratic 
form 2 ayx;x;, that is, by the directions of the characteristic vectors; 

j 


their Toagnitudes are such that a unit vector in the direction of the 
ith principal axis has its length changed to A;. In two dimensions the 
effect is shown in Figure 7.6.1. 

This was constructed for latent roots A, = 2, A, = ($) and charac- 
teristic vectors ¢p,, ᾧᾳ in the directions shown. Points (1), (2), (3), (4) 


Figure 7.6.1 
on the unit circle (a) are transformed into (1)’, (2)’, (3)’, (4)’ by the 


tensor. 

Next consider the effect of forming Ax, normalizing to form (AX), 
and then reapplying A. 

This is seen, from Figure 7.6.2, to result in a vector in which the 
component in direction @, is increased by a factor A,, and that in 
direction φς by A,. Thus the transformed vector is more nearly in 
the direction ¢p, than was originally the case. The sequence followed 
by successive applications of the process to a unit vector, initially (1), 
is shown. A(1) = (1)΄. This is normalized to give (1), A(1)n gives 
(2)’ and so on. It is clear that a few applications of the process will 
lead to a vector which lies, sensibly, in direction ¢p,. The figure also 
shows that, if an initial vector (0) had been chosen orthogonal to ,, 
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the operation A(0) would not lead to any convergence to @,, but 
merely to oscillation along τς. The whole argument can be general- 
ized immediately to an n dimensional hyperspace. 


be Unit circle 


Figure 7.6.2 


An alternative demonstration of the process just described, if 
A is symmetrical (i.e. aj = ἀμ). is as follows. The characteristic 
vectors are orthogonal, that is (¢p;.ep;) = Ὁ (i 7); we may there- 
express an arbitrary vector x as a linear combination of 

ep: 


X = Py + ἀρ, + « + + Pn. 


Now apply the operator A and use equation 7.6.1: 
AX = AAP, + AAgPy + ... + GnAnPn 


=i( aps + 405? 2 + pe 6 +45" φ,) 


so that, since A, > A, (n> 1) the result of the operation A is to decrease 
the proportion of vectors, other than q@,, in the system. Thus the 
pfocess converges, eventually, to A,@,. 

In practice it is usually simpler to take the vector x as having a 
maximum component unity instead of normalizing properly. This 
avoids the evaluation of (Σὰ) ἢ at each stage, and does not other- 
wise affect the process. 

When multiple roots occur, so that A, =A, = . . . =A, (say), 
there is no unique value of q@, and, under these circumstances the 
method breaks down. If, in practical application, it is found that 
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convergence is either slow or absent, it is well, before abandoning 
the process, to try an initial vector which is orthogonal to that 
originally taken, in order to eliminate the possibility that the first 
choice was at right angles to @,. 

The figure taken as an illustration in the preceding paragraphs 
was constructed on the basis of a positive definite quadratic form 
2ayxix;, for which all A; are greater than zero, The same argument 


is epplicable when some λὶ are negative, with the exception that 
the end points (1)’, (2)’ etc. will move from side to side of φς (assum- 
ing |A,| > |_| and A, negative). The normalizing process now consists 
not only of reducing the amplitude of Ax to unity, but also in 
restoring it to its original side of @,. 


For situations in which convergence is slow, various modifications 
of the above process are possible. In the first, instead of successively 
multiplying the arbitrary starting vector by A, some high power of 
A is first formed by the sequence: 


A, A®, AS, A®, A’... ete. 


Evidently, to form A*"requires n matrix multiplications, so that quite 
a small number of multiplications produce a high power of A. The 
iterative procedure then uses A®" instead of the simple A. It should be 
noted, however, that each matrix multiplication needs about $n° 
arithmetical multiplications so that detailed consideration needs to 
be given to work estimates before one method can be described as 
better than the other. 

Suppose that the ordinary method of iteration with A requires k 
steps and that A is an n x n matrix. The number of arithmetical 
multiplications required is thus kn®. If now, the power of A needed to 
produce the same result in one step is 2" we have that 2" = k and 
that the number of arithmetical multiplications needed to generate 
ΑἸ" is $mn*. 

It follows that, for the power method to be efficient, 


kmn3 < 2™n?* 
or m < 2™**In 


Some typical values are shown below. 


Order of matrix, π | 4 8 16 
m 59 25 26 
k = 2™ = number of multiplications 
required for simple iterative method 58 > 32 > 64 
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The values of k may appear too large, but when it is noticed that if the 
dominant and subdominant roots have a ratio A, say 0-9, to attain 
an accuracy of 10~-® in ¢, will, in general, need about 130 iterations 
with A, the advantage of the power method becomes obvious. 

Another method of improving convergence is to operate with 
(A — pl) rather than with A. Thus 


(4 — plop; = (A; — pip; 


so that (A — p/) has the same latent vectors ¢p; as A but has as latent 
roots (A; — »). Consider, for example, a matrix whose latent roots 
are + 3+ 2 and + 1. The rate of convergence of the iterative 
method with an arbitrary starting vector will not be better than a 
factor (8) at each stage. If, however, we iterate with (A — J) the 
rate of convergence will be proportional to (4) and if with (A — 3/) 
to (4), an improvement by a factor of 2 over the orginal. 

A third method of speeding up the determination of latent roots 
and vectors by the iterative method is to use Aitken’s δὲ process. 
Assume that three successive iterates up, u, and wu, are known and 
that u, tends to a limit in such a manner that the successive terms 
form an approximately geometric progression. In this event it may 
be assumed that: 


Ug = Ug + kr®™ 
Uy = Ugg + ἀπ τ" 
Uy = Ugg + kr"*? 
whence eT Po) Sa Mo 
Uy — Ugg Ug — Un 
or Uso = Ugly — Uj/Uy — Quy + tg 


It will be found that computation of the numerator of this expression 
usually involves differencing two nearly-equal quantities and thus 
implies high accuracy in the generation of the products uu, and 3. 
This difficulty is avoided by writing 


Pa et οἰ Β 
Uy τ Qu, _ Uo 


As an example of the power of the Aitken process, consider the 
matrix: 


lan = Ug — 


| — 5-509 882 1-870 086 0-422 908 
A=| 0-287865 — 11-811 654 5-711 900 
0-049 099 4-308 033 — 12.970 687 
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Starting with x, = [1,0,0]’, we find that: 


Xi Ax, X13 AX, ἄμ Axys 


1-000000 —17-35178 i:000000 —17-37848 1-000000 —17-38955 
—8-113909 141°12675 —8-133271 141-48389 —8-141326 141-63299 
7-873325 —137-09317 7-900812 —137-46826 7-910257 —137-62547 


It is seen that convergence is relatively slow but, applying the δ᾽ 
process to the values: 


— 17-35178, — 17-37848, — 17-38955 
we at once find: 
, ἜΒΡΝ = 17-3974 


which is a much better approximation to the true Amax = 17+3977 
than any of the original values. 


When A, is complex a variant of the above process is adopted. 
After a sufficient number of iterations it can be shown that 


A "x + bA"-1x + cA"®-*x > 0 
where ὁ and ¢ are numbers such that 
A? + bA+¢=0 
is the quadratic equation satisfied by A, and its complex conjugate. 


Thus, by choosing two values of n and remembering that vector 
equations are also satisfied by individual components, pairs of 
linear simultaneous equations are obtained which give ὁ and c, and 
consequently A,. 
page acted and possibly simpler, method has been suggested 
by workers at the National Physical Laboratory. This depends upon 
iteration with the operator (A + ip) where p is a suitably chosen 
When an approximation is known to any characteristic vector of 
a symmetric matrix, it is possible to obtain an improved value for 
the associated latent root by means of the expression: 
ΤῊ ΠῚ Te , 
L= (jx) ~ fet” ....«{7.0.2} 


Assume that 
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where the ¢p from an orthonormal! set and the € are small, scalar 
multipliers. Then, 


x’. Ax = (ep; + Lejp;)'A (ep; + Zejep;) 
j#i jai 
= (pi + Leweps)’ (Ap, + Zep) 
1 j#i 


= A; + Σεῖλ, 
jvi 


similarly {χ΄. x) = (<p; + 2are) (<p; + ZeEsp;) 
jai 
=1+ Ze; 
ji 
7 A; 
whence L=A(1+2 1 εἰ) [{ + ej) 
jai A jut 


= (1 +2 (Σ -- 1)e}) + O(e*) 


j#i 
so that the error in A; determined from JZ, is of order εξ, 


We may notice, before leaving the quantity L, that if x is any vector 
whatever, and A is a real symmetric square matrix, then: 

A, >L>”. ....(7.6.3) 

Having determined the largest latent root and its associated 
characteristic vector, it is possible to proceed to the determination of 
the next smaller root and vector. A crude method is to select any 
vector ,<p, orthogonal to ¢p, and then to go through the iterative 
process A,¢py, A( A, Pq), εἰο. Since ,¢p, is orthogonal to «p, we have seen 
that convergence to ¢,, will not occur, and instead the process will, in 
principle, tend to A,¢p,. Unfortunately, since the arithmetic processes 
used to form A,¢p, eéc. are subject to round off errors, a small propor- 
tion of ¢, will be introduced into the »,¢p, vectors, and this will grow 
until convergence to ¢p, instead of φρο, results. 

This defect is removed in an iterative method which is based 
upon the Gram-Schmidt process for the construction of orthogonal 
vectors. 

Assume that ¢p, is known, we then take any vector ,¢p, and form: 


ἄψφς — 24%); 
so that this vector is orthogonal to ¢p,. Now let: 
ὅλ. 2Ps = A. 1s — a4, 
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and repeat the process to obtain successively: 


gz- 3P2 = A. oP, — ας, 
arg. «Po = A. «τς — agp, 


. * - - - - * - - "-" 


wry -mP2 = A-m—1P2 — ἀηι--,Φ,. 


The multipliers .A,, λα, etc. are inserted in this scheme to show 
that the vectors stpo, stp, etc. are in standardized form. This standardi- 
zation may be effected either by true normalization so that the sum 
of the squares of the components of the ,@p. are unity or, more easily, 
by scaling so that the leading component is made equal to unity at 
each stage. 


The scalar multipliers a, a, etc. should be small after the first stage, 
and serve merely to keep cp, from contaminating the convergents 
to ᾧς. It is probably sufficient, in most cases, to insert an acp, term, 
not at each stage, but once every 5 cycles. The values of a,, a, etc. are 
determined from the orthogonality of τ; and each of the ,«p,, thus 


nAcP + νῷ: =0= ()-A-n_-1Po - On—1P)-P; 
whence On—1 = ΦΙ..4..,... γφρ, ἰφι 


When A, and ¢p, have been determined with sufficient accuracy, 
we chose a vector @, and form: 


δε. (ὃς = 4.γῳς — ap, — Bip, 
g3-aPs3 = A.sPs — αγφ; — βιῳ, 
mAs-mP3 = A-n—1P3 — On—1P1 — Bm-1P2- 


The values of αι, ay, 8, By etc. are again determined from the 
orthogonality of ~, and @, to each of the ,¢p,. Thus, 


"λα. Pi -nPs = 0 = PIA. n-1Pgz — ἀμ. Ὁ. Py 

nrg -P2-nPs = Ὁ = Prd .n-1P3 — Bn—1P2-Pr 
whence an) = DA. nn 1Po/P1-} 
Bu—1 = P24 -n-sPsleps-Po 


This determines A,, ᾧ 9» and a similar process with γῷς, dep, εἰς. can 
be used to obtain the other latent roots and characteristic vectors. 
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To check the values of the latent roots use may be made of the 
relations: . 
2 λι = tr Al 
r=1 


λυ λυ λα .8᾿;᾿ α «δ, — | A| 
which follow at once from the definition of the characteristic 
equation given in 7.2.8. 

The method just described has the disadvantage that it becomes 
progressively more complicated as successive latent roots and vectors 
are determined. A more satisfactory method is the following. Assume 
that A,,~p, are respectively the largest latent root and associated 
latent vector of A, and that ¢, is normalized so that p,.@, = 1. 

Consider next the matrix A, where: 


A, = A — AvP -P} ee . (7.6.4) 
and the product ,¢@; is defined as: 


φὶ φιῷ, PiPs - . .ὄ . PiPn 
PoPr P2 φφ,ς.... PolPn 


1PreP1 PunP2 PuPs- . .ὄ . Pu 
We have, from 7.6.4 


Asp, = Ap, — APi-Pi-P: | 
= Age, — λχφ) (since ¢p, is normalized and @,.¢p, = 1) 
== ἢ 
Again, if ¢p, is any other latent vector of A, 
Ap, = Ap, — AP. P1-, 
= Ag, (since the ¢p, is orthogonal to ¢,) 
- AP, 


Whence A, has the same latent vectors as A but the latent root 


corresponding to @, is now zero. 
The iterative method can now be applied to A, and will result in 


the determination of ep, and ἃς. A new matrix, Ay = A, — λιρ, φς, 


is now formed as before, this is easily shown to have the same latent 
vectors as A but to have latent roots of value zero associated with 
cp, and @,. The iterative method, applied to A,, will thus lead to 
A, and @,. Clearly the process can be repeated and will, in prin- 
ciple, lead to the determination of all of the A; and φι. 
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The «,; are not orthogonal when A is unsymmetric and the above 
method of removing roots is not available, J. H. Wilkinson has 
suggested that the following is the simplest procedure in this case. 
First we define row vectors 

ἂι = (dijo. - + Gin) 


such that the matrix A can be written: 


44122 ἄγῃ ay 
A =| %21%ee fan | =| % (765) 
GnjGnQ - ann an 


Next, let A, and @, be respectively the largest latent root and 
associated latent vector of A, assume that το, 15 the largest component 
of cp,. We now form a new vector x, which is simply ¢p, so scaled 
that its largest component @,; becomes unity, thus: 


Now, since latent vectors are undetermined to the extent of a 
numerical constant, 
Ax, — Ai) eee . (7.6.6) 


and, using the notation of 7.6.5, 


(from 7.6.6) 


whence, since by the definition of x,, x;; = 1, 
“yn =, ἐνόν [2»8.1] 
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and similarly, for any other latent root and vector 4,, x,, 
a,x, = A, ....[7.6.8) 


where it is assumed that x, is so scaled that x,; = 1. 
We now form the matrix: 


A, = A = x44; 
and notice that, by virtue of 7.6.7, 


i 
Ajx, = Ax, — ἘΠ Δ 


and, using 7.6.8 A,x, = Ax, — X,0;%, 
= Ax, — %4A, 
= A,(x, — x) 


or, since A,x, = 0, A,(x, — %) = A-(x, — %4) 


We have therefore shown that the matrix A, has latent roots 
A, . . . A, which are identical with those of A but are associated with 
latent vectors (x, — x,) (x3 — *;) . - - (¥n — ἃ) where x, is the 
latent root associated with the largest latent vector, A,, of A. The 
remaining latent root of A, is zero. 

The iterative method, applied to A,, will thus lead to the deter- 
mination of A, (the sub-dominant latent root of A) and to an associated 
latent vector (x, — xj). 

Now since (x, — x,) is, as usual, undetermined to the extent of a 
numerical factor, we cannot simply add x, (which is already known) 
to it in order to obtain x,. Assume, therefore, that the vector (x, — Δ) 
differs in scale from the x, already determined by a factor k. We then 
have: 


where the bracket is to be regarded as an operational symbol and 
must not be removed. Multiplying 7.6.9 by a; and using 7.6.7 and 
7.6.8 we have 

ἃς = Ay + kai(x. — *) 


whence xX, = χχ + — — (x9 — χα) ἐς (9.5.9) 
It should be noticed that the definition of A, implies that its ith row 
is zero. This, in turn, involves that all of the latent vectors of A, 
have their ith components zero. Thus it is sufficient to work with a 
matrix derived by striking the ith row and column from 4. The root 
removal process can be continued until all of the A, are obtained, a 
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sequence of matrices Aj, A, etc. being derived. The associated latent 
vectors can be obtained by the application of equations of the type 
7.6.9 in sequence starting from the sub-dominant root and vector. 


To conclude this section we may mention three other methods 
which have been used for the calculation of latent roots and charac- 
teristic vectors. The first is the so-called ‘purification’ process of 
RicHarpson (7), which seeks to remove the contribution of ¢p; from 
an arbitrary vector by multiplication by (A — A,J). This method 
demands a certain amount of art on the part of the user, and appears 
unsuitable for use on large systems and with automatic computing 
machinery. The second method is the so-called ‘Escalator’ of 
Morris (8), (59) which depends upon the solutions of a characteristic 
equation of order (n — 1) to obtain the characteristic equation of 
order (n). 


The operation of the escalator is as follows. Consider the quad- 
ratic form associated with a matrix A;,) of order n; this is: 
ει 
On = Σαρχρη. 
1,j=1 
Suppose that we consider the matrix obtained by removing the 
columr ἀμ (i= 1...) andthe rowa,; (7) ΞΞ 1. .. πὴ from A, 


, ἄγ μ»- ἄχη 


ἃ ἃ ἃ δ δ ἃ ὃ © αὶ  ἃ' 5 ἃ ἃ 


ον G-p'n-2 | ἄρνα 


. ἄμ) η--} Any 


FES ee SPST Fee STS Se 


Calling this matrix A(,_,) we have the associated quadratic form: 


QO, 1 = 2 ayxix; 
i,j=l 
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and we may write: 


n—1 


Qn = Qn-1 + *n Σ inks τξ' Xn Σ. Fa ἀν ee (7.6.10) 


If we assume that the latent roots = (r=1...n—1) and the 
characteristic vectors τ, = (¢r, dro . . - Sren—-y) (7 = 1... 2 — 1) 
are known for Aq, then it is well known, from matrix theory, 
that we can express Q,,_, as: 


n—1 
0... = 2s AD ~aeat Gb) 
r=] 


Now the components ¢,; of ᾧ, are in the directions of the original 
‘axes’ x; so that we have: 


(Dp = δι + gk y «eo bey pa n-1 ({Ξξ:] ... π-- 1). 
and the matrix Ly,—,), given by: 


. ἦν tt—1 
- Fe n—j 


ὃ ὃ ἃ ὃ δ ὃ δ δ ἃ ἃ ἃ ἃ ἃ 8 


is an ortho-normal one (1.6. it corresponds to a change from one set 
of orthogonal axes to another set related to the first by a simple 
rotation and without change of scale). Z;,,_,) is symmetric and obeys: 


Lin—1)-Lin-1) = I. 
We can thus express the quantities x, in terms of the @,: 
χ, = Lily + lage ᾿ . ἐν" ν Ὁ,--} (r = | ᾿ ..πη--ὄ Ὁ 
so that, using equation 7.6.11, 7.6.10 becomes: 
n—1 n-1l .-} n—-1 nal 
On = DADE + xn 2 in 2 letpe + Xn & Ong Σ᾽ ἱρῷ, + SnnXp- 
r= ἐπὶ = p=] j=l = 
. - (7.6.12) 


We notice, at this point, that x, is unchanged by the transformation. 
It is now evident that the matrix of equation 7.6.12, in terms of 
(Dy, De . . . Φ,...» Xn) is: 
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& Ὁ Biacenae 0 2 Gy 
=] 
ant 
a ee eee ὃ... Bac 
r=] 
ee ee ἐν * ypapey: yhechndh 
doa ἀμ aise oe a ae ta (7.6.13) 
eee ae, ee te SOAK BRP [ὁ a ets a 
0 Ἐπ μῶν πὰ ay a An—1 Py fe Eee 
r=] 
1 -1 
Σ᾽ Anrliy a Anrlor Σ᾽ Anly—y ann 


so that the characteristic equation becomes: 


|Aqy— AL | = λ, --λ 0 0 Ne ie Opn lyr τοῦ 


- δ Κ᾽ © ἃ ἃ ἃ © Ὁ © Ὁ δαὶ ἢ ὃ ἃ ἃ ἡ ἃ ἃ κα ἃ 


= = * ᾿ a cl * [1 - - - - ct = - - 5. Ὁ α ἃ ἃ ἃ 


..(7.6.14) 


where double suffix notation has been used to avoid writing out 
the summations. 
We now subtract a,,.1,,;/A, — A times the first column from the last, 


thus eliminating the first coefficient in column n, then yn. lgr/A, — A, 
times the second column and so on. The result is: 


0 A—-A O 0 0 
0 0 As—A 0 0 
ae : Ss ΟΣ “ 
δ 
Gugliy “Gules αωΐν . .. ἀρῖ,.χρ᾿ῤρ ἅμ, A (@nrlir) . (Grnlir) Grol 
= a ἄμε ae τα: 
pr 
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whence, expanding: 


Σ (Qnrlir) (@rnlir) 


amet free 
Bi mi x= 


or, since A and L are symmetric: 


n—1 
ἄρῃ —A= 2 P?/(Aj — A) ....(7.6.15) 
where 
n—1 
P,= Σ appli. ...+ (7.6.16) 
r=1 


Notice that P; is simply the sum of the products of the coefficients 
of the nth column in the matrix A with the components of the ith 
characteristic vector in its normalized condition. 

Morris has shown that the characteristic vector associated with 
any solution, A, of equation 7.6.15 is given by: 


Pry wie | i ε ἙΩΣ 
t= 2 bss Qs A) (r=1...2—1) ....(7.6.17) 


where ¢,,, is the rth component of the desired vector and /;, is the 
rth component of the characteristic vector associated with δι, 
the vector being in its normalized condition. Components, ¢a,, 
obtained from equation 7.6.17 have to be normalized to give the 
desired /),. 


The operation of the ‘escalator’ is as follows: 

(1) Solve a,, — A = 0 to give λ,. 

(2) Find ¢, associated with A,. 

(3) Use equation 7.6.15 to find A, and A, for next higher step. 

(4) Find ¢, and @, using A,, A, and equation 7.6.17. 

(5) Use A, and A, with ep, and @, to find A,, A,, A, and so on. 
The method when applied to an nth order matrix, leads to the 
intermediate calculation of about n?/2 roots and n*/3 components 
of characteristic vectors, so that the labour involved for large n is 
considerable. The method does not appear to be well suited to 
operation on an automatic calculating machine, but for hand com- 
putation for reasonable values of n (up to about 10) it appears very 


convenient, especially when the checks suggested by Morris are 
applied. 
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7.7 DESCENT METHODS FOR THE SOLUTION OF 
LINEAR EQUATIONS 
An alternative class of methods for the solution of sets of linear 
simultaneous equations depends upon the geometry, in hyperspace, 
of an associated positive definite quadratic form. 
Consider first the expression καὶ (equation 7.5.4). We have: 


S=}E.AE—D.E ΕΣ 


and we have seen, in section 7.5, that the surfaces § = const. repre- 
sent a set of hyper-ellipsoids whose common centre has co-ordinates 
(ξι =, ἔχ = χα. . . ἔμ =x,) which define the solution of the set 
of simultaneous equations 7.2.1. 


Figure 7.7.1 


The problem of obtaining a solution of the original equations may 
now be reduced to the following equivalent: given any point (Ep), 
find a process which will lead to a point (€,) for which the value 
of S is less than that for (&). A repetition of such a process will 
eventually lead to some point (&,) from which no possible decrease 
in the value of § is possible within the limits of accuracy assigned. 
The point (&,) is then the ‘solution’ of the equations. 
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We shall now show that an infinite number of such processes is 
possible, and shall discuss some of the more significant variants. 
First take any plane I" passing through the point (&) and contain- 
ing the residue vector (ry) (see section 7.5). The surfaces δ᾽ = const. 
will intersect this plane in a set of ellipses having a common centre 
Cy (Figure 7.7.1). 

Consider any vector (v,) lying in I” and making an angle > 7/2 
with the direction of (r,), then, for points (€,,) on this vector and 
sufficiently near to (€,) S will have a smaller value than that at (ἔξ). 

The mode of procedure is clearly to fix upon some (v,) and pro- 
ceed along it until S no longer decreases, at (&,) say. To determine 
(E,) we notice that at (&,), (v,) and the residue vector (r,) are at right 
angles (notice that r, will not generally lie in J"), and this condition 
may be ensured by making the vectors v, and r, have a zero scalar 
product. 


Now any point (&) on (v,) may be written: 


E=& + av, .4.4{{7..28} 
where a is a scalar, Again, by definition: 

r= ΑἘ -- ὃ (7.7.3) 
whence 
and consequently: 

ΓΞ +aA.v,. .....(7.7.4) 


Now at (r,) we have (r,.v,) = 0 whence, from equation 7.7.4 : 
or 
a = — (Fy.0,)/(v,.A0,) ....{7.7.8} 


so that our end point (&,) is given by: 


Be = be — gy ο εν.) 


So far, the vector (v,) has been completely arbitrary [except for 
the general condition of making an angle > 7/2 with the direction 
of (r,)], which demonstrates the truth of our original assertion 
regarding the infinite number of possible descent processes; we 


now examine some of the more practical aspects of the choice 


Three choices of (v,) have significant value: 
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(1) (v,) is taken parallel to an axis having unit vector (x,). 

(2) (v,) is taken in the direction of steepest descent from (E>)— 
that is at 180° to (r,). 

(3) (v,) is so chosen as to pass through C,. 


The first choice gives the so-called ‘relaxation’ method for the 


(χε. ἄχ Ὁ 
Tok 

--- ae * δέρ. -“"5 κα © (7.7.7) 
ake 


That is, the component of &, in the direction of the x,th axis is 
decreased by r.,/a,z. Notice that by r4, we mean the residual from 
the kth equation at the first approximation. 


The virtue of the relaxation method is the simplicity of the 
formula given in equation 7.7.7. Its defect lies in the fact that x, 
has to be chosen by means of some ill-defined criterion, usually by 
considering the equation which gives the largest residual. In actual 
computation a skilled operator will often operate on several values 
of χε at once, a process known as ‘block’ or ‘group’ relaxation, 
Unfortunately, a high degree of skill is required to make full use 
of the method in its widest sense, but, even so, the most inexperienced 
operator can always obtain a result eventually. 

For use with an automatic computing machine it is desirable 
(even if not mandatory) to use a process which calls for no decision 
based upon the judgement of the user. Such a method is the ‘steepest 


descent,’ for which we take v, = — ro, and thus obtain: 

, : (Fo-To) 

δι = ee --- (r,. Ar) Fo sas . (7.7.8) 
1} 
Σ᾽ 19% 

τς - — - Fo 
Σ᾽ Ais oi -T oj 
t, j=l 


(ro is the ith component of ry). 

For well-conditioned equations this method is excellent; when, 
however, it is applied to an ill-conditioned set, the situation shown 
in Figure 7.7.2 may arise, in which the minimum §, slightly overshoots 
a principal diameter of the ellipse system. 
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Under these circumstances the solution may oscillate from side to 
side of the principal diameter and, in consequence, require many 


Figure 7.7.2 


iterations to reach Cy. Experience has shown that more rapid con- 
vergence may be obtained, for ill-conditioned systems, by taking: 


Θ(Γο. ΓΟ) 
Bi = & -- ER "αὶ To »eee(7.7.9) 


for, say, four iterations and then making a single application of the 
exact formula 7.7.8. 

The third method of procedure may be derived as follows. 
Suppose that, at the previous stage of the process, there was derived 
a vector v, and a point on it, &,, such that (r,.v,) = 0, just as was 
done in the previous derivations. We now have a pair of orthogonal 
vectors v, and r,,. Consider the system of ellipses δ᾽ = const. in the 
plane of v, and r,, shown in Figure 7.7.3. 

Now any vector in the plane I’ containing r, and v, can be 
written: 

On +1 = aly + Bov,. ....({7.7.10) 


1 Cana is the centre of the ellipses S = const. then, at Cran the 
residue vector Fr, is perpendicular to I’, and consequently to both 
r, and U,, so that (fa41-Tn) = (Γι. Ὁ.) = 0. 
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Again: 
Ents = Sn + Une, = En +4Fn + βυ,. cneel thal 
Figure 7.7.3 
And since 


r= AE — ὃ. 
Tata = Tn + A(En+1 — Sn) = Γρ +aArn + βάυ,. ..(7.7.12) 
But, from the orthogonality conditions: 
(Ta-Tno1) = ὃ = (ΓΝ. Γι) + (tn. Ary) + B(Py-AUn) 
(Oy .Ta+)) =O = a(v,.AT,) + B(v,-.Av,) 
(v, and r,, are ab initio orthogonal so that (r,.v,) = 0) 
whence: 


(Tn -Tn) (Un- AUp) 
* (Dn. Ar,)? — (Fa-Afn) (Vn. AVp) 


B ἘΣ = (Γ, Tn) (On- Ap) 
© (On Atn)® — (fa At n) (On-AOn) 
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The set of formulae (7.7.11-7.7.13) may be regarded as the two- 
dimensional analogue of the steepest descent formula (7.7.8); they 
have the advantage that the solution is obtained in exactly the same 
number of steps as the equations have variables. This is due to the 
fact that each new steepest descent vector is orthogonal to all pre- 
vious descent vectors. Thus, since the hyperspace has only n 
dimensions, only n such vectors exist and the process is completed 
at the nth vector. In this respect the method is closely related to the 
Gram-Schmidt orthogonalization procedure. 


7.8 A CAUTIONARY PARAGRAPH 


We have mentioned, in various places, the fact that the residue 
criterion based upon: | | 
S= ξξ.κξ --- δ. 


assumes that the matrix A is both symmetric and positive-definite. 
It is worth reminding the potential user of 7.7.6-7.7.13 that these 
equations must only be used when A is known to obey these 
conditions. 
A cautionary example is provided by the equations: 
x, — 2x, + 3x, Ἔχ, = 3 
— 2x, + x, — 2x, —A, a —4 
3x, = 2x. + X53 + DX, = 7 
Xy — χη + 5xg + 3x, = 8 
for which the solution is (1, 1, 1, 1). An application of equation 
7.7.8 from the starting point &, (0,0,0,0) for which 


r, = (— 3, + 4, — 7, — 8) 
leads to 
Ey = (+ +34, — -45, + +79, + +90) 
with 
= ( 1:51, + -39, + -21, — -56) 


which is quite satisfactory [(rg.T)) = 138, (r,.r,) = 2-79 ]. Afurther 
application of the procedure leads, however, to co-ordinate changes 
of large magnitude O(10) and to an increase in (r,.r,) to 1475-49! 

The reason for this behaviour is clear, since when A is not positive- 
definite, the expression S represents, in general, hyper-elliptic- 
hyperboloids, and the geometrical assumptions made in section 7.7 
no longer hold. 


110 


DESCENT METHODS BASED ON R* 
7.9 DESCENT METHODS BASED ON R? 
By means of the positive definite form: 
= (r.r) = (AE — 6)(AE — δ) ....{7.5..1} 
we may construct the analogues of the descent formulae previously 
derived for S, but with.the advantage that, since R? is known to be 
positive-definite, this limitation on A is removed. As a penalty for 
this the formulae are, of course, slightly more complicated. 
We again choose an arbitrary vector (v,) and seek to find the least 
value of R? when some multiple (av,) is added to &. Thus: 
Rie αὐ) = [AE + av,) — 6]. [A(E + av,) — δ] 
= (AB. AE) + a°(Av,.Av,) + 2a[(AE.Av,) — (6. Av,)] 
— 2(6. AE) + (0.6). 
At the minimum we have 
dR? 


a =8 
whence: a(Av,.Av,) + [Av,.(A—R — 6)] = 0 
= = — (ἅυ;. το) /(Av,.Av,) 


which is easily shown to give a minimum value for R*. It follows that 
the correction formula corresponding to equation 7.7.6 is: 


E=&— Gea εὖ. sain eh Ree) 


Just as in the previous case we may choose v, in a number of 
ways, if it is taken parallel to an axis having unit vector Χο, we obtain 
the relaxation formula: 

is (Ty. Axx) 
ft = So — (ax “πὴ Ἢ 


or, in terms of the matrix elements 


..(7.9.3) 


ἐπ} 


The steepest descent method is obtained by choosing (v,) to have 
the direction in which ἐξ, in the region of ἔν», is changing most 
rapidly. This is, in vector notation: 


= — grad R? ....[7.9.4] 
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where the éth component of Ὁ), v,, is given by: 


ὃ 
mk = — δε, (5. 1. ..(7.9.5) 


Now an inspection of the quadratic form of R?, given in equation 
7.5.3, shows that: 


ὃ 2) = ᾿ Σ == 2). 
aca 


n 
= a oii »+++ (7.9.6) 
whence 
grad (R?) = 24'r ree Vee 


where A’ is the transpose of A and r is the residue vector. We thus 
obtain 
(7). AA'T5) 


δι = bo — (aan, (Ado) 4% .++0(7.9.8) 


which is the analogue of equation 7.7.8. 


Finally, we may find the minimum of R? over a two dimensional 
plane containing any pair of non-parallel vectors. It is convenient 
to choose these to be (grad R?) at &, and the vector v,, used to derive 
En from &,,;—this pair being orthogonal. Thus: 


Sn +1 = Sn + Onts ....(7.9.9. 
Uni, =a(grad R?), + Bo, = 2aA'r, + Bo, ....(7.9.10) 


where a and β are determined so as to make R2(é,4.,) a minimum. 
Using 7.9.9, 7.9.10 and the definition of R® 7.9.1, we have: 


R%Eq41) = (AEn+1 — δ).(4ξ, κι — δ) 
= [AE, — b + aA(grad R*), + BAv,). 
(46, — 6 + aA(grad ΚΞ) + BAv,) 
= (tf, + 2aAA’r, + βάυ,). (τὰ + 2aAA'T, -+ BAD,). 
Now we require, for a minimum: 
ORT an OR 
da op 
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whence: 
4AA'r,,. (τ, + 2aAA’r, + BAv,) = 0 
2Av,,.(T_ + 2aAA'r, + BAv,) = 0 
giving: 
2a = 
(r,,-Av,)(Av,.AA'?,) — (T_.AA'Ty) (Ap. AUn) | 
(AA’r,. AA'T,)(AU,.AV,) — (Αὺ,. ΔΑ’ τι) Αυ,. AA'Ty) 
ΡΒ. | 
(r,-AA'T,)(AU,.AA'T,) — (Tp. AV,)(AA'T,. 4.4’ τῷ) 
(AA'r, . AA'T,)(AUp,. Αυ,) — (AU,.AA'T,) (AV,. ΑΔ’ τι) 
These formulae are far too cumbersome for use by a human com- 
puter, but may possibly find application with an automatic digital 
calculator. 


vee e(7.9.11) 


7.10 A NUMERICAL EXAMPLE 

To illustrate the merits (and otherwise) of the preceding methods, 
we give their application to the set of equations proposed in section 
7.8. It has already been pointed out that methods based upon S will 
fail for this set of equations since A is not positive-definite. 

Using the relaxation technique of section 7.9, based on R? and the 
initial vector ἔν = (0, 0, 0, 0), we obtain Table 7.10.1 by a direct 
application of equation 7.9.3. 


Table 7.10.1 Relaxation on set of equations of section 7.8 


k ἔξ Γὼᾳ ξ ry fy Ig és ΓᾺ ἔς Γι 


o=5 [ὅ τῶ δ ἜΝ ὃ = es] ὃ -ἰηὴρ 
0.40 1217 0 41-235) .432 + 1-667] -432 + 1.335 
0 —7 | 0 2:17} -466 - 2:6831) "466 + 1-767|-632 + 1-933 
0 —8 |1-83 —2-50|1-833 —-171| 1-833 — -603| 1-833 + +227 
(ro.%o) = 138\(r,.r,) = 17-00) (τς. γε) = 8.53 (γῳ. γ9) = 6°67 (γι. γε) = 9°59 


The residues which indicate the value of & (i.e. the co-ordinate 
number) in equation 7.9.3 have been underlined. We notice that it 
is not always possible to select the co-ordinate of largest residual 
at each step since, as happens in k = 4 of r, a relaxation with respect 
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(r,.7_) τὰ 138 | (r,.4;) = 2°72 (r_.%_) = -79 (γε. Γι) = "68 


i = 
ἐς — *0122(A'r,) 
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to this vector has just been performed so that no further improvement 
can be produced by further alteration to x, at this stage. 


Next we give the results of the same number of steepest descents 
applied to the same set of equations. 


Table 7.10.2. Steepest descent 


ἐς Ὁ 18 5 @0 | & » Cal & 5 ἐδ ἃ 


0-3 —40 +488 +1-416 + 2380] +-302 +-488 - “605 4+-°384 +-366 —-603 | +-426 +-478 
0 +4 +32 —390 207 —1-863 +°301 +°612 —1-292 | +409 +-709 - 280 | . +439 + -578 
0-7 —64 +781 +-050 —-246 | +-+884 +-413 +1-163 | - 88 +-182 —-623 | +-911 +-216 
ῦ ..8 —66 +°805 —-802 —1-007 | +1-227 +-102 3.347 1.108 —-097 - 218 | 4+-1-177 - 083 


ἔν τῷἬ = = 


ἔν + -419(A'r,) ἔν + -0136(A'r,) ἔν +°069(A'r,) 


It will be noticed that, initially, the convergence of the steepest 
descent process is faster than that of the relaxation, but that from &, 
there is an oscillation of the type indicated in Figure 7.7.2. This is 
to be expected in the present set of equations which are quite ill- 
conditioned. 

Finally, Zable 7.10.3 shows the results of four descent to ellipse 
centre applications based on equations 7.9.9-7.9.11. Since &, is the 
same as a normal steepest descent and is given in Table 7.10.2 and 
since v, = (Α΄ τὸ) and r, are also given in the latter table, we do 
not reproduce them here. 


Table 7.10.3 
(A’r,)=|+ -230| (AA’r,) =]4+2-211| Av, (= AA'r,)= | — 362 


— 1-863 — +824 + 306 
— 246 — -865 — 578 
— 1-007 | —2°158 | — 590 


(r,.AA'r,) = + 4-60, (AA’r,.AA'r,) = + 10-973, 
(r,.Av,) = + 13-39, (Av,.AA’r,) = + 720-67, 


whence: 
2a = — -441, B = + -000336 
v,=(—-114] ἔ,Ξ 2. -374]) re=[+-316] (τς. κω) = -70. 
+. "833 + +443 +. +734 
|} + -086 + -867 + +238 
4. +429 4-1-2297 —- 0535 
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In a similar manner we obtain: 


v0;=|+ -080 = 5] 
+ +136 
+ -109 
— -074 

and: 

v,=[+-097| ἔξ, ΞΕ -551 re=[4+-198] (γι. τῷ = "40. 
+ +102 + -681 + +541 
— 005 + -971 — -258 
— -057 + 1-096 + 019) 


7.11 MATRIX ‘ROTATION’ AND LATENT ROOTS 


Probably the simplest methods of determining both latent roots and 
latent vectors of a matrix are those in which the original matrix is 
transformed into some form from which the roots and vectors can be 
derived directly. The first such method is due to Jacobi and was 
re-discovered by von NeuMANN and Go.pstineE"®) in the late 1940s. 
In this method the matrix is transformed into truly diagonal form 
by means of an orthogonal transformation matrix. 
Suppose that A is the matrix whose latent roots and vectors are 
required and that 7'is an orthogonal transformation matrix so that: 
Tl ὦ Τ Τα] mer ie 
Now, 7’(A — Al) = (T’AT — Al) from 7.11.1 whence the A, 
which are the roots of |A —A|=0, are also the roots of 
|T’AT — A| = 0, that is, the latent roots of 7’A T are also those of A. 
Now let (A;,y;) be respectively a latent root and associated latent 
vector of 7’AT. Then by definition, 


T’ATy; = Ai 
or, pre-multiplying by 7, and using 7.11.1, 
ATy; = A: Ty; 


whence | 
(A — Al) Ty; = 0 


which shows that Ty; is the latent root of A which corresponds to λι. 
The Jacobi method produces a matrix Τ᾽ which has the property that: 
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dy 

doo 0 
T'AT=D= 
0 
dun 
from which, by inspection, 
Ay= dy, = (1,0, 0,0, . . . 0); Ay = ἄρρεν Ξε 0, 1, 0,0, . . 0); 
. Ay = dy ete. 


Tis determined as the product of a sequence of elementary trans- 
formations, C,,, each of which produces a matrix in which the largest, 
off-diagonal, element of the preceding matrix is reduced to zero. 

The basis of the method is the fact that (see section 7.5) the latent 
roots of the matrix A are the squares of the inverse principal axes of 
the quadratic form: 


Fk sa = 1] (ai; = aj) me AL Re 
The orthonormal! matrix T has the property that: 
z= T.x. 
transforms 7.11.2 into 
Σλιξῖ Ξε 1 opnnbe ean) 
i= 1 


where the A; are the latent roots of A. It is well known" that ifC is any 
orthonormal! transformation matrix so that the transformed variables, 
y are given by 


y=C.x (c’.C=1) 


then the matrix of the transformed quadratic form in y, corresponding 
to 7.11.2, isC’. A.C. It should be noted that the sum of the squares of 
the diagonal elements of C is greatest when C = T, that is, when the 
transformed matrix of coefficients is diagonal. 

The Jacobi process determines T as the limit of a sequence of 
simple rotations involving only two of the axes (x;,) at a time, the idea 
being to eliminate the largest (a,;) from A (i τέ 7) and then the next 
largest, and so on. 

Suppose that a, , (= 4, ,) is the largest off-diagonal element of A, 
and that we seek to eliminate it by a rotation of axes through an 


116 


MATRIX ‘ROTATION’ AND LATENT ROOTS 


angle 6, in the plane of (x,, x,). The equations of transformation are: 


x, = sin@,.y, + cos 6,.y, + ox (FAS 
χιξενι (τε τσ, 5) 
and substitution in equation 7.11.2 gives for the coefficient of 


(Yrs Ys) 


x, = cos @,.y, — sin a | 


2 cos 6, sin 0, (a,, — @s,) — 2 (cos? 8, — sin® @,)a,, 
so that to eliminate this we take: 
tan 20, = 2α,.[(α,, — ss) oe {7.11.5] 
which defines the angle of rotation (6,). 


Now the transformation matrix C, say, corresponding to equation 
7.11.4 is: 


| (TALS) 


me νον ere ee 0 
rth row -Ὁ cos θ᾽ 00... 0 -—sin@, 
ἃ Aa. 10 
sth row >] 0 0. sin 8,00. . Ocos θ᾽ 


* - - - * * * = - ἃ ὦ δ ἃ ἃ αὶ αὶ ἃ δα ὃ δ ὃ δ Β΄ 8 


 Ξ *« ἃ ἃ ἃ ἢ ἃ ἃ ἃ ὃ δ ὃ ἃ ἃ ὃ ὃ & ἃ 


t ft 


rth column sth column 


and it is easily proved, by direct substitution, that the sum of the 
squares of the elements on the diagonal of the transformed matrix: 


A, = CiAC, 
is simply: 
2a?, + Σ aj cso 81.) 
i=] 


which shows, since it is greater than 2 ajj, that A, is nearer (in some 
tut 
sense) to the required diagonal matrix than was A. 
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A repetition of the above process, with the largest off-diagonal 
element of A, will lead to a new transformed matrix Ag, thus: 


δ ὃ ὦ δ ἃ ἃ ἃ 


élc. 


where A,, tends to the matrix (A;) and the continued product 
C,.Cy.C, . .. tends to the diagonalizing matrix Τ᾽ defined in 
equation 7.11.1. 

The Jacobi method has the advantage that all of the operations 
involved are very simple. Convergence is rapid in suitable cases but 
it must be remembered that, in principle, the method is one of 
successive approximation and not a finite step one. A minor dis- 
advantage, from the viewpoint of the automatic digital computer, 
is the fact that the off-diagonal elements must be scanned at each 
stage to determine which is greatest. 

Starting with a suggestion of Lanczos"!” in 1950, new methods of 
latent root and vector determination have been developed in which 
the matrix is not diagonalized but is reduced to triple-diagonal form. 
It turns out that transformation to this form can be made in a finite 
number of steps and that the latent roots and vectors of the triple- 
diagonal matrix can be fairly easily determined. The two best-known 
methods of transformation are those of Grvens"*® and Hovuse- 
HOLDER'!#), 

In Givens’ method co-ordinate transformations are again applied 
but this time so that the element reduced to zero is @,_;,s(r > 1) 
and the transformation, corresponding to 7.11.4, has: 


tan 6, = Gp 1,8/4-— lr 


It is no longer necessary to determine the largest off-diagonal element 
and the elimination proceeds systematically with the liquidation of 


413, ἄγχι.) γεν . - . Gin 
a4, as, # ᾿ ΓῚ Qoy 
Qn_-9n 


The method has the advantage, over that of Jacobi, that the ele- 
ments reduced to zero remain null so that the labour of transforma- 
tion decreases as the work proceeds. It has been estimated by 
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Wiixrnson‘®) that the work of transformation to triple original form 
is only about one-twentieth as long as that involved in an equivalent 
reduction to diagonal form by Jacobi’s method. 

The second method of reduction is that of Householder. We define 


T = Ty-1-Tn—3 . . . Τὶ, for an nth order matrix A. Here 
Tm = 1 — Qwytn 
where wit, = 1 
and - τὰ (0.0....... Ὁ es ἄρρεν ον SD 
whence x, = 1 
iam 


Starting with T, the matrix: 


is formed with such a choice of xg, x3 . . .ὄ x, that the elements 
a,, of the first row of A are reduced to zero in A®), Next A® 


ἄμ so 
is found from A‘) = T;A®)T with a choice of the x, . . . Xn to 
make the elements ag, . . . ἄρῃ zero in A“). The process continues, 


with increasing ease, until A“~” has been found. This is in triple 
diagonal form. 

The defining equations, the requirement for zero coefficients in 
the transformed matrix, and the consideration that the sums of 
squares of elements in any row of the transformed matrix must be 
equal to that before transformation are adequate to evaluate the x; 
at each stage with the exception of a sign which is so chosen as to 
ensure maximum numerical stability. 

WILKrnson"®) has shown that, for the 4 x 4 matrix: 


4 bb ὦ 
A ΜΗ by by ἔῃ 
Gy ἕῳ Sy 


the optimum choice of values for (0, x9, *3, Χ4) 18 


I | eC) __ €5(b;) i d,s(b;) 
saa 2} μή χε ao 2xo+/s "Beals 
where s = 6? - εἶ + dj and s(b,) means ‘sign of by’. 
The number of multiplications required in Householder’s method 
for reduction to triple-diagonal form is about §n* compared with 
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$n° in Givens’ method and 2n* in Lanczos’. Furthermore, Givens’ 
method requires the extraction of ἐπ square roots compared with 
the 2n of the Householder process. Both methods can be extended to 
cover the ease of unsymmetric matrices although here the reduction 
leads to almost triangular rather than tri-diagonal form. 

The last phase of the determination of latent roots and vectors by 
the Givens or Householder methods is the determination of the actual 
A; from the triple-diagonal matrix. Several possible methods for 
doing this exist“” but the simplest conceptually and the most stable 
numerically, is the following. 

First note that the matrix is assumed to be: 


0 
elc. 


as 


Denoting by p, (A) the rth leading principal minor of (T’AT — AJ) 
it is easy to see that: 
βο(λ) = ] 
ῥβι(λ) = a, -- A | 
px(A) = (a, — A) (ας — A) — BS 


»ᾳἢἾ δ ἃ δ & © δ δ)Ὺ᾽ δι:  Ξ΄ῥ  δὃῇὲῪ΄᾽ δ᾽Ὺ7Ὺ᾽ὃ᾽Σ2  δ' δ "ἡ LJ 


pA) = (a, — A)pr—1(A) = Bipr- 2(A) 


It can be shown that, if when any #,(A) is zero its sign is assumed to 
be the opposite of that of p,_ (A), the number of agreements in sign, 
a,, between consecutive numbers of the sequence f(A), /,(A), 
bo(A), . . . Pa(A) is equal to the number of latent roots of T’AT 
which are greater than A. 


To use the method a pair of numbers (a, δὴ) is found such that 

σα 5 K>o,. The value of ogy, is then computed. If o, > σα.» 

By ie ' i: wl 

then a root, A, lies between a and Stic if, however, o44..> 0, the 
ΕΗ 
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root lies between “Ἐ" and 6. According to which criterion is 
satisfied, either o1/; a+) OF o1/a+ ,) 18 computed and the test again 
s(e3-) Ξι 1 9) 


applied. It is seen that the precision of location is increased by a 
factor of 2 at each partition and that the convergence is independent 
of root separation and magnitude. 

To start the process several strategies exist. For a complete deter- 
mination of roots, a determination of A,,, by the iterative method 
followed by a survey of the values of the p,(A) for lesser values of A 
will give the rough positions of roots and this can be followed by a 
precise determination of any desired roots by the method just out- 
lined. 

On the other hand, a rough bound for Ama, may be found as follows. 
If x = (x,,%9, . . - Xn) is any latent vector of 7’AT and AQ is its 
associated root, then, since 


T'ATx = Xx 
oe + a% + Bra i%r-91 = Ax, 


Assume that x, is the numerically greatest component of x. We then 
have the inequality: 


IA] < |B] + Jarl + [B41 


andasurvey of p,(A) for values of A between + (|8,| + |a,| + |8,+1]) 
and — (|8,| + |a,| + |8,.,|) must entrap all of the latent roots. 

The accurate determination of the associated latent vectors, once 
the latent roots are known, is still a non-trivial problem. Wilkinson 
suggests that the best method is as follows. For the latent root δ, 
calculate the matrix (7’AT — A,J). Reduce this to an upper tri- 
angular matrix U; (say) and then solve the equations U;x = ὁ, where 
εἰ = (1,1,1, . . . 1). The solution x; is then the required latent 
vector. 


7.12 MONTE CARLO METHODS 


The somewhat intriguing title of this section is now applied to 
methods of numerical analysis which make use of the theory of 
Games'!®), The suggestion that such methods might be applicable 
appears to have been due, originally, to voh Neumann and Ulam; 
but statisticians have since claimed that the techniques are simply 
those of small sample analysis and, as such, have been known for a 
considerable time. 
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Monte Carlo methods appear to be applicable to most problems 
involving linear operators, and the present example of a technique 
for determining the elements of an inverse matrix, was the first 
actual procedure to be worked out. The reader familiar with the 
game of solitaire will recognize the basic similarity. 

Suppose that it is desired to form the inverse of a matrix A, of 
order n. We first form the matrix: 


pe eae τε {5.1.1} 


and assume, for the present purpose, that the relation: 
[1 — Ayw(A)| = |Ay(E)| < 1 ee Ae 


holds, λμ being the greatest of the latent roots in the respective 
matrices. Now subject to equation 7.12.2 it can be shown ™® that: 


AA (f— BA =f4S+R...4P4...<28 
om ἢ 
Pe 
whence, formally, 
τ 
(A), = Σ (Εἶν .... (7.12.4) 
k= 0) 


where (),; represents the element of the ith row and jth column of 
the matrix ( ). 

We now consider a ‘random walk’ of the following type. Let P), 
P, ... Py, be a set of (n) points, then we start from any point 
P; and jump from point to point in such a way that the prob- 
ability of a direct move from P, to P, is p,,. At any point P, there is 
a probability p, = 1 — 2'p,, that the walk ends there. Now, at each 

ml 
transition P, — P,, there is a transition value v,, defined by: 
δ. «Ῥω τὸ. ὃς ΠΕ Αἰ νον 


where ¢,, is the corresponding element of the matrix FE. If we define 
the value of the ‘ walk’ to be: 
Ww, = 0 if the walk ends at any point P, where k τὸ j 
scsi Vii, Vii, ++» + Uj,~-Pj; ' when the walk ends at P; 
where the initial value is unity and the points of the walk are P;, P;, 
.. .P;, Pj, then the expected value can be shown'®.2)) to be 


(A-"),;; that is the (1)th element of the required inverse of (A). 
This follows from the relation: 
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(Expectation) ;; 
oO a Σ Σ 
- 2 4 cn ὦ CG CG e+ OF 
OS ee i= 1 ij=1 int" if kd 


...+.(from 7.12.5) 


oo 
= My t+ E (By = Σ (By = (Ag 
= = & . (from 7.12.4) 


where δὲ; is the Kronecker delta function, and arises from the fact 
that when i -- 7 there is a probability p; that the walk will at once 
stop at P;, and that the transition value is, in this case, pj '. 
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8 
PARTIAL DIFFERENTIAL EQUATIONS 


8.1 DEFINITIONS AND SCOPE 


In ΤῪΗΙΒ chapter we shall be concerned with the numerical solution 
of certain simple types of partial differential equation. In the past, 
it is true to say that only the linear types have received much atten- 
tion, chiefly because of the technical difficulties of the calculations 
involved. The present era of high speed automatic digital computors 
is doing much to remedy this, but the storage capacity of these 
machines is still too limited to make possible an attack on the more 
complex types of equation. 

The partial differential equations of mathematical physics often 
fall into the form: 


ew aw ew a? 3 a 
A ae +a ee +B ae +2655 +2F +0 Gn =! 
: ....08.1.}} 
in which A, B, Ο, F, G, H, I may be constant (including zero) or 
functions of (x, γ, 2) (5 μα =) Thus, when 
* \Ox dy ὃ: ᾿ 
A=B=C=1, H=G=F=/]=0 
we obtain the Laplace equation: 


ΟΝ ow, ΟὟ 
Ox2 ᾿ ὃγ52 ᾿ 8.3 
If J = const. 8.1.1 becomes the Poisson equation, and if W is a 


function of x and y only, these equations reduce to the well-known 
two dimensional forms. In two dimensions, if 


= 0. 


H*® — AB>0 
the equation is said to be Ayperbolic, if 

H*? — AB=0 
parabolic, and if 

H* — AB <0 


elliptic. ‘These terms are borrowed from the analogous quadratic 
forms, and are related to convenient subdivisions into which the 
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solutions naturally fall. The methods of solution resemble strongly 
those which have been discussed for ordinary differential equations, 
especially in their relation to the boundary conditions which the 
solution has to satisfy. We shall examine, first, simple equations of 
parabolic and hyperbolic type which are similar in the nature of 
their solutions to one-point boundary condition equations in the 
simpler case of ordinary differential equations. 


8.2 PARABOLIC AND HYPERBOLIC EQUATIONS 
IN TWO VARIABLES 


The classical example of a parabolic equation represents the one- 
dimensional flow of heat in a conducting wire, or the diffusion of a 
liquid or gas along a porous tube, and a study of this equation, from 
the finite difference viewpoint, lead Courant, Friedrichs and Lewy 
to their fundamental theorem on the relationship between the space 
and time intervals. The usual form of the diffusion equation is: 
ew aw 
picid 

of aa δι “οἴ 8....}} 
with initial condition W = f(x) at t = Ὁ, and two boundary condi- 
tions of the general type W + a — = b(t) atx =1,, andx = ἰς, say. 


We will assume that the equation is to be solved by replacing 
both of the derivatives by finite difference approximations at 
intervals (5x) and (δι) respectively. Using the simple relationships: 

OW δ, _ W(x + dx, ἢ) — 2W(x, δ) + W(x — 8x, ἢ 
dx? ~ (8x)? (Sx)? 


....(8.2.2) 


aW __ AW _ W(x, t +81) — W(x, ἢ (8.2.3) 


Bia” aa (82) 

we obtain, by substitution in equation 8.2.1: 

W(x, t + δὴ) = yW(x + 8x, ἢ + (1 — 2y) W(x, t) + yW(x — Sx, ἢ 
.. ++ (8.2.4) 


and: 


where y = c*8t/(8x)*. κὸν oeeiod 


Equation 8.2.4 enables W(x, ndt) to be calculated for any value of n 
by successive build up via 


W[x, (n — 1)8t)] W[x,(n — 2)8¢] . . . W(x, 0). 
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If the value of W(x, t) is required at some particular time T we must 
sg πδὶ = T 


which may be considered to determine the interval (δὲ) in terms of 
the total time T and the number of steps (n) required. Thus: 

y = c®T /n(6x)?. 
In a like manner (6x) must be an integral sub-multiple of the length 
of the (x) boundary. 

The important contribution of Courant and his co-workers was 
to show that it is not possible to choose (8x) and (δὲ) arbitrarily if 
a stable solution is to be obtained. By considering the difference, 
e(x, ἢ), between the solution of the differential equation 8.2.1 and 
that of the difference equation 8.2.4, it was shown that this error is 
bounded only if y < 4, and that it grows exponentially with ¢ when 
y > 3. It follows that, when solving an equation of the general type 
8.2.1, (δι) and (6x) must be so chosen as to make ¢*Sz/(Sx)? < 4 and 
that, in consequence, unlimited decrease in the value of (8x) will 
not lead to improved accuracy unless accompanied by a suitable 
decrease in (δὲ). 

‘Milne has shown that if W(x,t) has continuous partial derivatives 
of order 6 in x and of order 3 in #, then the difference between the 
true solution of equation 8.2.1 and the solution of the difference 


equation 8.2.4 satisfies: 
c®T (dx) 4 (Se ) 
mar 


le(x, ἢ] < “τές (a ....(8.2.6) 


where the value of the partial derivative is taken in the (x, #) region 
covered by the solution, and the optimum value ? is taken for y. 

An alternative procedure to the above is to replace only one of 
the partial derivatives by a finite difference approximation and thus 
obtain either: 


d? | 
ε3 dx? [W(x, t + δὴ + W(x, ἢ] 
4W(x, ἢ 
(δι) 
which is an ordinary differential equation in x for W(x, ¢ + δὲ) in 
terms of the known values W(x, ¢), or: 


ππι [W(x + 8x, t) — 2W(x, t) + W(x — dx, t)] = wi t) 


(dx) | 
Βα @ (8.2.8) 
which is an ordinary differential equation in ¢. 


= δ [W(x, t+ δὴ) + W(x, Ὁ)}] — εν ++ (8.2.7) 
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The methods described in Chapter 6 can be applied to solve these 
equations which present no special problems. It may be mentioned 
that the process defined by equation 8.2.7 is often called the 
‘Hartree-Womersley’ method, 

A typical hyperbolic equation which occurs in practice is the 
wave equation: 

aw δ᾽ 

ὡ ~ ἃ 

This time the central difference formulae for the derivatives may 
be applied to both sides with the result: 


W(x, ὁ +- δι) = BL W(x + 8x, t) + W(x — dx, ἢ — W(x, t — δὴ} 


....(8.2.9) 


Ἐ2(1 —B) W(x, ἢ ον 10] 
«[διλ3 
where p=a(s). (8.2.11) 
2 
Thus, by taking B = c? (x) = 1, equation 8.2.10 reduces to 
W(x, t + δὲ) = W(x + 8x, t) + W(x — dx, ἢ — W(x, t — δὴ 
....(8.2.12) 


which has exactly the same solution‘, W = f(x + ct) + g(x — ¢t), 
as equation 8.2.9. It follows that solutions obtained by replacing 
equation 8.2.9 by 8.2.10-8.2.12 will be exact. This procedure is 
particularly useful in dealing with the subsequent motion of a 
vibrating string whose initial form and velocity are prescribed. In 
this situation the numerical solution, via 8.2.10—-8.2.12, is consider- 
ably easier to compute than the results of the exact analysis using 
Fourier series. 

The methods just described can be applied in largely unmodified 
form to parabolic and hyperbolic partial equations of more com- 
plicated types, with the exception that the derivatives which occur 
must be replaced by suitable difference approximations. In a less 
measure the same remark applies to non-linear equations, although 
here great care is needed to ensure stability. The reader requiring 
further information is referred to the excellent paper of Βιιανοηίϑ), 


8.3 HIGHER DIFFERENCES AND CHECKING 


A possible method of obtaining a more accurate solution, for the 
same intervals of differencing, might appear to be to use ap- 
proximations to the partial derivatives involving higher orders of 
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differences than those considered in section 8.2. This procedure, 
unfortunately, often leads to the generation of spurious detail in 
the resulting solution because, in effect, the original equation is 
being replaced by a difference equation of higher order. A less 
dangerous procedure is to use higher difference formulae to estimate 
the error of a result obtained by the use of the simpler methods. 

Thus, in place of equation 8.2.4 we may obtain a more exact 
relationship via the central difference formulae in equations 4.2.10, 
4.2.11 of Chapter 4. 


δα all ἐνὸς + aeBt—ete)W(y) (8.8.1) 


ow 
= By [(μδ)ι — ἐδῥίμδ), + ψοδέ(μδ), — τξοδέ(μδ) ΠΡ, ὃ 
..(8.3.2) 
where ὃ, δὲ indicate that the differences are to be taken for constant x 
and constant ¢ respectively. It should be noted that, in making the 
comparison, the optimum value: 
y = c78t/(8x)? = ἃ .....(8,3.3) 
should again be used. 


8.4 EQUATIONS WITH MORE THAN TWO VARIABLES 


The extension of our previous finite difference treatment to dimen- 
sions outside the (x, ¢) plane presents no special difficulties. We shall 
illustrate the method of approach by considering the two dimen- 
sional Laplacian operator: 


V*W==- += : .....(8.4.}) 
A fairly obvious start is to replace the partial derivatives by their 
finite difference mr eek 


ἔπ ΠΡ = δε !ψ (δα) = a ERE Re I) ANI) tS Oe) 


are 83W/(8y)? = δ᾽ ᾿ [W(x, » + δ») —2W(x, ») + W(x, » — δ) 
. (8.4.2) 
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whence: 
νὴ» = [ W(x =f 5x, 9) = W(x -- dx, 7) “ft W(x, 9 + 5] 
~~ (8s)? + Μία,» — dy) — 4 W(x, ») 

. .(8.4.3) 
where we have assumed that a square difference grid of side 
ds = 5x = ὃν is being used. 

Equation 8.4.3 can be pictured more vividly by the point pattern 
shown in Figure 8.4.1 (a), which is conveniently represented by the 
matrix Figure 8.4.1 (b). If we remember that the Laplacian, Ὁ 3, of 


a ds ss 
(a) (b) 
Figure 8.4.1 


a function W, at a particular point in space is independent of the 
particular co-ordinate system chosen, equation 8.4.3 suggests that an 
alternative expression could be obtained by rotating the axis system 
through 45° and working in terms of the corresponding points in 


-2 A 


ds" 
(a) (6) 
Figure 8.4.2 


this system. Inserting the function values in terms of our original 
co-ordinate system, and noticing that ds’ = 6s 4/2, we obtain: 
VW= 
[ W(x — ὃχ,γ + Sy) + W(x + 8x,» + δ») ] 
2(8s)* L+ W(x — 8x, » — ὃ) + W(x + dx, » — ὃ) — 4W(x, ») 
. (8.4.4) 
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The preceding arguments have been merely suggestive and we 
now propose to investigate more closely the nature of the approxi- 
mations involved; for convenience of writing we shall represent the 
expressions formed from the function values shown in Figure 8.4.1 (b) 
and Figure 8.4.2.(b) respectively, by H and 2X (‘Horizontal’ and 
* Cross’). 


We first introduce the operators E, and Εἰ defined by: 


E, [W(x,9)] = W(x + 6x, ») } 
Ey [W(x, ¥)) = W(x,» + dy) 


with symbolic associations, coresponding to equation 3.1.11 of 


Chapter 3, 
ota an 
From the expression for H we have: 
H(W) = (E, + Ez" + E, + Ey" —4)(W) 
or, using equation 8.4.6 and the exponential expansion theorem, 
H(W) = (85)? Μ᾽») + ve (δ)" Va(W) + soo (8s)° 55 ΟΣ 


..(8.4.5) 


sth a"w 


where V}?(W) τὸ -- - + or: 


We thus see that a more correct 


form of 8.4.3 is: 


| ἘΠ 
VW = Be ΗΓ) — Ο[(δὴ). ....(8.4.8) 


Again, from the expression for 2X, we have: 
2X(W) = (Ἐφ Ἐν + E,.E, + Ez'.E,* + E,.E,' — 4)(W) 
which leads to 
a W) = 2(δ)" VW) + ξ(δ)" V8(W) 


᾿ ἑ 
τὴν Va) + SP τόσα VW + ο.. 


. . (8.4.9) 


+ (as) & iB) = lee 
whence: 
VW) = raat [2X(W)] — O[(8s)2]_ .... (8.4.10) 
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and we notice that as is to be expected from the effectively larger 
interval of differencing involved in calculating (2X), the error term, 
although O[(5s)?], has a larger coefficient than in equation 8.4.8. 

In the special case of the Laplace equation where V*(W) = 0 
we have 


ow aw ow | 
i. a lu $V (WH) 
so that equation 8.4.9 becomes: 
(δι)" 


2X(W) = 2(δ)" ὋΨ)) — ξ(δ) 602») + Te5 Vo(W) 
whence, using 8.4.7: 


(4H + 2xy(W) = δ(δη" Ὁ) + &2 yyw) 


or VW) = sagt (4H + 2X)W — O[(8s)4]. .... (8.4.11) 
The operator (4H +- 2X) is easily remembered as: 


ek ρος. ee. Ϊ 
", 8 3 . 5 ΞΞΞ 4 —20 4 ΞΞΞ Κ "-μ"»5᾿.α (8.4. 12) 
Ι ἃ ἃ 


the symbol K being suggested by ΜΠΙΝΕ (3) who, by introducing the 
symbolic operator N?, defined by: 


| 1-2 1 74 
N2?=2X—2H= -- ἘΠΕ τε + OLE) 
Ege a2 
. (8.4.13) 
shows that 7? = 
l ( Κ' Κα KN? ~~ Κι K2N2 N& 


6(8s)2 \* — 79 + 3940 — 180 — 120960 * 3780 — 504 


. . (8.4.14) 
The foregoing analysis enables us to obtain each senandiets 


for solving the two-dimensional equivalents of the diffusion and 
wave equations considered in section 8.2. 
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Thus, corresponding to equation 8.2.1 we have: 


had - ew .ν..(8.4.15) 


and obtain at once: 
W(x, γ, ὁ + 8t) = (8t)e°V2W + W(x, », ἢ. 


We now replace \7*W by a suitable finite difference approximation; 
if the range of (¢) over which the solution is to be continued is not 
large it will be sufficient to use equation 8.4.8. Thus: 


W(x, 9, t + δὲ) = c? ---. δε 


Aya ~HW+W 


and, if the optimum value y = c? ant = 4 is chosen, this becomes: 


W(x, y, 1 + δὴ) = (4H + 1)W .... (8.4.16) 
or, symbolically: 


Ϊ 
W(x, y, - δὴ = 3%) 1 2 1] W(x, 9,2). ....(8.4.17) 
l 


If greater accuracy is required we may use equation 8.4.11 and 
εἶδ 
thus obtain | again with y = =; = + |: 


W(x, y,t + δὴ = (4K +1)W=4; W(x, y, t). 


.. (8.4.18) 
The matrix (K -+ 36) is peculiar in that it can be expressed as: 


πρὶ 


which, as mentioned in a paper by Yowell, quoted by Milne (loc. cit.), 
enables the iterative process defined by equation 8.4.18 to be 
adapted to calculation on punched card or other fixed scanning 
sequence computing machines. 
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The two-dimensional wave equation can be similarly treated, 
thus: 


—o ety Ἢ +». (8.4.19) 
becomes, on replacing the differentials by differences: 
t 
Ws, 8+ δὴ =F ED KW (a, 351) + 2H, 8) — Ws 8) 
. . (8.4.20) 


Milne has shown that, for convergence, we must take 


B = c®(8t)?/(8s)? < 
and proposes the value β = 3, he thus ae (actually by using 
equation 8.4.14 to replace ἡ ἢ 


Ws »,1- δὴ = (Te — iggo) 258) ἘΦ (e358) — Wat —B) 

+ O[(ds)*] .««.(8.4.2}} 
K? is to be obtained by two successive applications of K, a procedure 
which avoids trouble at the boundaries, 

Hyperbolic and parabolic partial differential equations are closely 
analogous to one-point boundary condition ordinary differential 
equations, in that all the information regarding the solution is known 
initially, so that the solution can proceed on a step-by-step basis. We 
have not considered initial conditions of the more complicated types 
involving first derivatives of the function with respect to space or 
time co-ordinates, but these can be introduced without any great 
difficulty, and the diversity of types and situations would make too 
long an account for the present purpose. In the same way we have 
considered only hyperbolic and parabolic equations of the most 
elementary kinds; the method of approach to more complicated ex- 
amples is still via finite difference approximations, but the conditions 
for convergence are far more difficult to ascertain. Often, in situa- 
tions representing real problems, a knowledge of the physics of the 
system will give a clear indication of any marked inaccuracy in the 
solution; in the absence of such guidance, however, the best that 
can be done is a check on the solution, as indicated in section 8.3, by 
means of a more accurate approximation to the equation, using 
differences of higher order than those used to obtain the solution 
originally. It cannot be too strongly remarked, however, that such 
higher difference formulae should not be used in the original calcu- 
lation, since they are equivalent to replacing the original differential 
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equation by one of higher order, and may thus introduce spurious 
detail into the solution. A better method is to decrease the size of the 
interval in the ‘open’ direction, although here again care must be 
taken to compensate, if necessary, with a decrease in other intervals. 
Unfortunately, more general versions of Courant’s results are not 
always available but the known forms may give some idea of the 
dimensions involved. 


8.5 CLASSIFICATION CHARACTERISTICS AND CANONICAL FORM 
It was mentioned in section 8.1 that partial differential equations of 
the type 8.1.1 were divided into classes according to certain criteria; 
we shall now show how these criteria arise, and how they are related 
to the canonical forms to which the equation may be reduced. 


ΩΣ 


Figure 8.5.1 


In the first place we introduce the symbols: 


ow _oW 

P= Ox μ᾿ 
Ow - ΘΗ ee ΘΝ 
ΓΑ δ δῶν "δ 


which are common in the theory of surfaces. In terms of these 
symbols equation 8.1.1 may be written: 


Ar + 2Hs + Bt=@ ἐν ετἴ8.5.}} 


where ® may be a function of the remaining quantities in-8.1.1. 
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Now the following relationships obtain: 


ape 2 2 ye 4 .«.{8.0}.} 


Ox dy 
ὃ ὃ 
ἀφ = 5 ἀκ + 5 dy = sdx + tdy. νοι δὴ 


Next assume that the values of we p) and (= 4) are given 
on some curve J" in the (x, y) plane (Figure 8.5.1). Equations 8.5.1- 
8.5.3 thus constitute a set of simultaneous equations for the deter- 
mination of (r, s, ) and will have a determinate solution only if: 


dx dy 0 
4=|0 dx dy|#0. 
A 2H B 
The equation 4 = 0 or: 
dy\? dy a 
4(2) —2H(2)+B=0 ΠΥ 


defines two sets of curves. These will be real and distinct only if 
H? — AB> 0; only one curve will exist if H? — AB = 0, and no 
real curves will exist if H2 — AB < 0; the curves are known as the 
‘characteristics’ of the given equation. 

If the two families of characteristics corresponding to the solutions 
of 8.5.4 are called: 


φίχ, γ) = const. and f(x,y) = const. 
it may be shown “) that the substitution : 
E+ in =4(x,y),  -- ἰη τ, ψα,)) ὠ ----(8.5.5) 


reduces equation 8.1.1 to the canonical elliptic form: 
2 2 | 
aw on a=x(m ὅς a ἮΒ Ὶ νιν. (8.5.6) 


8ὲ5 + δη3 ᾿ δὲ ee 
whilst the substitution: 
&=¢(x,7), η = (x7) 2 ORT) 


gives the canonical hyperbolic form: 


om = X(W, Ge - ee .). 2 (8.5.8) 
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The parabolic form is obtained when: 


ἕξ = (x, 9) = (x,y); 7-4 +++ +(8.5.9) 


MULTI-POINT BOUNDARY CONDITIONS 


These equations may be rearranged into the matrix form 


4 2 --| 0 O-------O]|y, 79 
and is Ber | Bi aad Geis τως ree 0 Ve 
ew /. aw ee ee, τ eee eee 0}. 0 
δη" — χ(νν, δὲ 95 ἕ . ). eee « (8.5.10) Oe ς ee δ ee ee ς δ = an a ΝΣ . (8.6.3) 


Characteristics thus specify, in some sense, a set of co-ordinates in 
which the general equation reduces to the simpler canonical form. 
It thus becomes appropriate in certain cases to take as a grid for 
finite differences the characteristics themselves. This technique 15 
likely to be useful only in simple cases in which equation 8.5.4 can 
be handled by direct analytical methods; it has found some applica- 
tions in aerodynamics.‘® 


or A.y = ὃ say, whence, if we can find A~', we can obtain the 
solution to 8.6.1 as y = A-.6. 

Now, in the case of the matrix A of equation 8.6.3 the inverse is 
easily shown to be: 

A= 

(1 —1/n) (1 —2j/n) (1 —3)n) (1—4/n)-- ὀ ὠ2|π I/n 
(1 — 2/n)2(1 — 2/n)2(1 — 3)n)2(1 —4/n) - -- 2.2/n 2.1 /n 
(1 — 3/n)2(1 — 3/n)3(1 — 3/n)3(1 —4/n) -- - 3.2/n 3.1)/n 
(1 — 4/n)2(1 — 4/n)3(1 — 4/n)4(1 — 4/n) - --, 4.2/n 4.} [ἡ 


—= 
mm — a i ὧν a a eS τ Ὁ 


8.6 MULTI-POINT BOUNDARY CONDITIONS AND ELLIPTIC PARTIAL 
DIFFERENTIAL EQUATIONS 


Although it may appear curious to include a discussion of ordinary 
differential equations in a chapter on partial equations, this position 
is really the most suitable for those of the former class which have 
to satisfy multi-point boundary conditions. The essential identity of 
the principal method of solution for elliptic partial differential equa- 
tions and that for the ordinary variety will, we hope, be clear after 
reading this section. 

We will start by considering the simplest possible ordinary 
differential equation of the second order: 


y’ =0 (8.6.1) 


and we will assume that the solution has to satisfy the two-point 
conditions (y = 3%, x = 0) (y = Jn, ἃ = ndx). Now equation 8.6.1 
can be approximated by: 


δῆ, = 0 (m=1,2...n—1) 


so that we obtain as our finite difference solution to equation 8.6.1, 
subject to the given boundary conditions: 


(1 — 1/n)yo + Inn ....(8.6.8) 


(1 - 2|πῈ)ν + 2ynjn 
sal — 3|π}}0 + 3ynjn 


(y) = 


— — — = — — — -- —= 


ΤΣ ἅτ = = ὦ ΙΒ τς 


thatis by the setofsimultaneousequations; me i (itstsi‘iC [|| στΠτττ΄- 
γε τ Wit Is =0 (1 /n)y%o + (n — Vann 
Jy — a+ Is = 0 Since the analytical solution of equation 8.6.1, subject to the 
J2— Wo t+Ia = 0 8.6.2 boundary conditions, is 
linn Sd bebe rely Sees se acc _ (In — Jo) 
‘aula aaa vd 


In—3 — μον + Pn-1 = 
Jn-2 — 2yn-1 + jn, = 0 
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we see that the solution given by 8.6.5 is, in this case, exact, 
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The same method can be adopted for the more general equation: 
J IR) 


with similar boundary conditions. In this case the set of difference 
equations becomes: 


82ym = f(mdx). (Sx)? (m= 1,2...2—1) 
and the solution may be written: 


(y) =| γι | =A. |», —flSx) (8x)? | ow 0 (6.6) 


Ms — f(2Bx) (Bx)* 
- — f(38x) (8x)* 
ἥρωι yn — τ -- ηδα](δι): 


where A~! is the matrix defined by equation 8.6.4. 

These simple examples will give the idea behind a general method 
for solving multi-point boundary condition differential equations. 
The solution proceeds in three parts: 


(1) Represent the given equation by a finite difference approxima- 

tion of the same order. 

(2) Set up the system of simultaneous equations defined by the 

finite difference approximations at each point (méx) [m = | 
. . » (2—1)] at which a solution is required, using the 
boundary conditions to define y, and y, in the case of second 
order equations; for higher orders the boundary conditions 
may define yo, y, εἶδ. . .. In—1 Ine 

(3) Solve the simultaneous equations to give the required solution 

of the original differential equation. 

It is evident that the fortunate accident of A having a known in- 
verse will not, in general, happen, so that the set of simultaneous 
equations, corresponding to 8.6.2, will have to be solved by one of the 
methods described in Chapter 7. Much existing work in solving two- 
point boundary condition differential equations has been carried out 
by the use of the relaxation method (Chapter 7, sections 7.7, 7.9), and 
the computational layout for such cases has been well described by 
SouTHWELL.”) When a high speed automatic digital calculator is 
available, however, modern practice would favour the use of pivotal 
condensation (Chapter 7, section 7.3). As a check in the work, a 
finite difference approximation of higher order may be used. This, 
however, must not be made the basis of the original solution. 
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Non-linear equations can be solved by the same method, but since 
these will lead to non-linear simultaneous equations the work of 
solving the latter may be prohibitive. 

A natural extension of the preceding discussion allows the 
solution of elliptic partial differential equations. We shall consider 
first the most common form of these—the Laplace equation in two 


dimensions: ΟΜ =0 .....(8.6.7) 


with W = Wyonsome boundary Γ΄. Τὸ start with, it will be assumed 
that I’ consists only of elements parallel to the co-ordinate axes (x,y). 


~+---- 
tn 

᾿ ---;-- 
Ι 
Ι 


~ 


Figure 8.6.1 


We divide the interior of I’ over which a solution to equation 8.6.7 
is required, by means of a co-ordinate grid of equispaced lines, and 
call the interior points of this grid, 1,2 . . . 8 (Figure 8.6.1). At each 
such point the equation 8.6.7 may be approximated by H(W) = 0 
(8.4.8) so that, for the region shown in Figure 8.6.1, we obtain the 
set of relations: HW) =0 (ei...) 


and, if the boundary values are B, . . . By as shown, these become: 


4W,— W, - Ww, = B, + By 
—W,+4W, -- Ws = B, + Bs 
4W,— W, = B, + By + Bro 
= μὴ — Ww, + 44, -- W, — μ᾽ = () 
— Νὴ — W,+4W.— We - W,=0 
᾿Ξ W,+ 4W, = B,+ 8, + Β, 
Ws +4W,— W,=B, + By 
- W, — W,+4W, =B, + Bs 
....(8.6.8) 
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In matrix form this set of equations may be written: 


G.W=b .. (8.6.9) 
where: 
ὦ ea ἔυοΣ δ Ὁ ὃ Ob ....(86.610 
4. Ὁ ὃ - ὃ ¢ 6 
"Ὁ ῦ ὃ Ὁ. Ὁ 
Sf ἃ οὐ δ ΞῚ 0 
—J G =| Set ODO =] 
ὃ ὅ ὃ .ΔὉΞ ἃ. Ὁ 9] 
QO Ὁ =] δ Ὁ 4 —] 
θ. ὃ 6 —— .9ϑ .Σ ἃ 


and 6 is the column vector whose components are the right-hand 
elements of equations 8.6.8. It is evident, from the form of equations 
8.6.8, 8.6.9, that the matrix G is independent of the boundary values 
themselves, and also that G is always symmetrical. Further inves- 
tigation (8) shows that G is non-singular, and has latent roots which 
lie in the range 0 < A; < 8, these roots being symmetrically located 
with respect to A = 4. 

The solution of the Laplace equation 8.6.7 subject to the given 
boundary conditions is therefore approximated by the solution 
to 8.6.9: 


W =G-1.b .. ++ (8.6.11) 


so that the methods of Chapter 7, and in particular the relaxation 
method, can be used to solve this problem. If a solution to the 
Laplace equation is required for the same boundary shape but a 
number of different boundary values, it may be more expeditious to 
evaluate G~! and then to use equation 8.6.11 rather than to solve 
the given equations by relaxation each time. 

Exactly the same method can be applied to the solution of 
Poisson’s equation: 


ΝΡ = p(x, 9) ....(8.6.12) 
except that the individual difference equations will now become: 
H(W,) = (8s)*p, 


where p, indicates that the value of p is to be taken at the lattice 
point corresponding to W,. 
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Thus, in the example of Figure 8.6.1, the resulting equations are: 


G.W =|B, + By + (85)2p, ]  ....(8.6.13) 
B, + Bs + (85)*ps 

Bs + By + By + (35) *ps 

etc. 


B, + Βς + (85)*ps 


where G is the matrix of equation 8.6.10. 


We now consider the question of lattice dimensions and boundaries 
which are not straight lines. In the first place, it is clear that (8s) 


a We Wei 
{ 3 
(ds) "5 ‘5 
m3 
"2 

5° "ἢ 

ΤΈΡΕΟΞ 1 1 
2 

“5 | 

3 

sro duc ? ! 
4 
“5 9) 

awe 14 1 
. ἢ ὰ 

"5 

1 Ι 


--...... a 
(a) [δ] (c) (d) [6} 
Figure 8.6.2 


= 
| 
— 


must be chosen so that variations in boundary shape and value can 
make themselves felt on the solution. Thus, with the mesh (3s) shown 
in Figure 8.6.2, boundaries (a), (δ) and (¢) and boundary values (d), 
(e) and (f) will lead to identical solutions of the finite difference equa- 
tions approximating 8.6.7 or 8.6.12 although it is quite evident that 
the true solutions are by no means identical. It is clear, therefore, that 
theinterval (8s) must be sochosen as to allow adequate representation 
of the detail of thesolution as revealed by the known boundary values. 

On the other hand, physical considerations, for example, of the 
related problems of charged conductors, suggest that far from the 
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boundaries much of this detail will be lost, so that an overall 
decrease in mesh size would be wasteful in computing time. To 
overcome this difficulty it is usual to take a small mesh size near to 
sharp discontinuities and to increase this towards the body of the 
region of solution. This is illustrated in the enlarged view of the 
region B, By, By of Figure 8.6.1 given in Figure 8.6.3 where it is seen 
that a mesh of different size has been adopted in the region of the 
discontinuities. 


Bios Fioe 8; 


bots 


——— i 5 


By SS a τς εἶχαν : καῖ μα νας ἦν πῶς ὅκα ἅττ τὶ πεῖς 


Figure 8.6.3 


The joining up of such small meshes to the main one is carried 

out in steps; for example, in Figure 8.6.3 we should: 

(1) Evaluate the solution using only the coarse mesh shown in 
Figure 8.6.1. 

(2) Use linear interpolation from (1) and the boundary values B, 
Bo, Byos to find approximate values of W at (a) and (6). Thus 
Way =4(W, + Wa) Woy =t(W, + Wir + Ware + Wares): 
Similarly for Wi. . . Wey. 

(3) Apply relaxation or other techniques to refine these approxi- 
mations, noticing that alterations in Wa)... Wy) will 
produce changes in the main net W, — W, so that this will 
require further treatment at the same time. 
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(4) When adequately small residuals have been obtained for 
Wea) . a (f) W, eee Ws; add the next finer mesh 
(a) (B) (y) etc. and proceed as before. 
As the intervals decrease near the boundary it should be found that 
less and less alteration is required in the main structure. ; 

The above types of procedure are well adapted to straight 
boundaries of the sort illustrated, but with curved boundaries, and 


e2 


ῳ «3 
Ps 5e<—ds —+e! e2 
ds 
“ a 
3 4 
6° ν΄ 
(a) (b) 


Figure 8.6.4 


for straight boundaries which suggest hexagonal or related structures 
+t is sometimes more convenient to adopt one of the finite difference 


approximations“); 
3W,—W,—W,-W,=- a (85) °p, 
for the triangular mesh shown in Figure 8.6.4 (a), or 
6W, —W,—W,—W,—W,—We— Μὴ 
= — $(55)"[p. + ὑς (δε) *pi] 
for the hexagonal mesh of Figure 8.6.4 (0). 


....(8.6.14) 


εν. .(8.6.15) 
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We may remark, in passing, that it has been recommended that, 
in decreasing the mesh size, a two stage process of the type shown in 
Figure 8.6.5 should be adopted: 

First the large grid, one element of which is (2, 4, 8, 9), should 
be refined. Then the diagonal grid (2, 3, 4, 5) should be adopted, 
the value of W, being taken as }(W, + W,-+ W, + W,). After 
refinement of this the small grid (1, 2, 7, 3) may be taken with the 
value W, estimated as 3(W, + W, + W, + W,). In our experience 
this cautious approach is not usually justified, and the direct process 
of halving the mesh edge at each stage is to be preferred. 


=——— πτῖτες τα ἡπῖσπ 
—=——_— — ἔτ ΒΕ 
——= τι πα 


—- ...-...-.- o_o  δίριν cae 


nA Ὁ Ὁ πτἱ 


The general treatment of curved boundaries is difficult; formulae 
can be constructed to introduce the boundary values in a general 
situation such as that shown in Figure 8.6.6 for which 


VW= 


2 W(x + 859, 2) me W(x; 3) W(x = ὃς, y) ae W(x, 2) 
ὃς + 85, | ade 2 ὃς ] Ξ 


2 EK +s,) — W(x, 9) r W(x, y — 855) — W(x, 3) 
d5; + δὲ 1 OS 
— O(6s) .... (8.6.16) 


and from which it is seen that the error 15 considerably greater than 
is the case with the symmetrical formula 8.4.8. Asymmetrical 
formulae can be constructed (10) to have an error O[(ds)?] but they are 
very complicated and we prefer to adopt a locally finer grid in the 
boundary region and use equation 8.6.16. 
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8.7 PRACTICAL ASPECTS OF THE RELAXATION METHOD 


It seems likely that the large-scale solution of the simultaneous 
difference equations which result from attempts to solve elliptic 
partial differential equations will soon be a process which is ex- 
clusively performed on high speed computing machines. ‘The best 
method of solution in this case is one in which a standard procedure is 
followed in all examples, as, for instance, in the elimination method 
of section 7.3 or the steepest descent methods of sections 7.7, 7.9. 


Occasionally, however, it will always be necessary to solve given 
problems by hand and in this event the relaxation method seems 
to offer the simplest procedure and, at the same time, the greatest 
scope for intuition and experience on the part of the human com- 
puter. We therefore propose in this section, to mention briefly, one 
or two practical aspects of relaxation technique as applied to the 
solution of equations involving the Laplacian operator. 

To illustrate the method we shall examine the two dimensional 
potential problem represented by the system shown in Figure 8.7.1. 

In one physical interpretation of this situation A and B represent 
the pole faces of a magnetic recording head and C represents the 
recording medium; all are assumed to define equipotentials. 

In obtaining a numerical solution of the Laplace equation 


V2¥ =0 
in the air gap region of Figure 8.7.1 a difficulty arises at once from 
the fact that the air region extends to infinity in the directions 
shown. This is overcome in practice by assuming that at some 
finite distance the solution becomes indistinguishable from the 
solution at infinity. For the system shown this is the linear function: 
Veg = 100y/g 
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term which means simply that all of the values of V are changed 
simultaneously and in the same direction until the residuals are 
reduced as far as possible. 


at infinity in a direction parallel to C, and the similar function: 


at infinity in a direction parallel to A and B, the » axis being taken 
to lie midway between A and B, 

For the present purpose we will assume that we are interested in 
a solution of accuracy | unit of potential, and experience suggests 
that the field will be sensibly that at infinity when OX = 5g and 
similarly for OY. We thus replace Figure 8.7.1 by the system shown 
in Figure 8.7.2. 


γε -100 y/g γε 100 ¥/g 
—- + 
x 


Figure 8.7.2 


Next we notice that the distribution will be anti-symmetrical about 
OY so that OY is at potential zero and Figure 8.7.2 can be reduced 
to Figure 8.7.3. 


Finally we observe that the solution will be symmetrical about 
OGZ (making an angle of 45° with OX) so that our computations 
can be confined to the region OGLX. 


The next consideration is the choice of a finite difference grid 
upon which to carry out the calculations. Normally, for the accuracy 
required, we would take a mesh of side 5s = g/10 but for the pur- 
pose of this example we shall start with a very coarse mesh (g/2) 
and decrease this progressively to illustrate the technique. 

We thus have the system shown in Figure 8.7.4. 

The mesh points concerned are (a . . . i) and we might start by 
taking the initial values of V at these to be zero. If the residuals are 
now calculated [taking them to be H(V) where H is defined by 
Figure 8.4.1] we find H(V,) =0, H(V,) ... H(V,) = 100, and 
H(V,) = 150. Now a situation of this kind in which a number of 
adjacent mesh points have large residuals of the same sign is usually 
treated by a technique known as ‘block’ or ‘group’ relaxation, a 
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V«100y¥/g 


SouTHWELL™») has given a physical picture which is often help- 
ful in seeing what sort of movement is required in block relaxation. 
He shows that the solution is equivalent to finding the equilibrium 
position of the points a . . . i if they were connected together, and 
to the boundaries, by means of light elastic strings. The boundaries 
are assumed to be at levels (perpendicular to the plane of the paper) 
proportional to their potentials. 
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Thus, for Figure 8.7.4, a more plausible starting point would 
be to take the ‘levels’ of a... at height 50, ic. midway 
between those of the two boundaries. If this is done we find 
H(V,) = — 150, H(V,) . . . H(V;) =0 which is considerably 
more satisfactory. 

The first relaxation net is now as shown in Figure 8.7.5 which also 
gives the inscriptions which are inserted as the relaxation proceeds. 


Figure 8.7.6. Relaxation No, 1 


It is our practice to write the initial value under each lattice point 
and then inscribe residuals in a vertical column to the right of the 
point; corrections, if any, are written on the left. One point of pro- 
cedure may be mentioned: remember to correct residuals at adjacent 
points after altering any lattice-value, neglect of this obvious warning has 
often led to error. Numbers are always chosen so that only integers 


. . 8 
ἂ [1 [1 . 
{a) fa) [8] (a) 
- [1 . 
{1) (2) 
Figure 8.7.6 


appear on the computation sheet, this means that it is not worth 
reducing residuals to below 2 units in magnitude. At this point the 
corrections are added into the initial values and the residuals 
recalculated to ensure removal of casual errors, then the process is 
restarted with the residuals multiplied by 10 or 100. 

For the present example we do not yet perform this scaling up 
of residuals, we first divide the mesh interval by 2, obtaining 
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Figure 8.7.7, and then refine this as far as possible. The procedure 
for obtaining initial values at the new points is shown in the figure, 


γ8 
| 50 
5 


= ° 
= 


0 
Figure 8.7.7. Relaxation No. 2 


first those values marked (a) are derived by taking } x sum of 
the adjacent net points (and boundary values if necessary) obtained 
in the first approximation as shown in Figure 8.7.6 (1). When all 
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(a) 


(a)-type points have been evaluated these are used, with the original 
values, to obtain intermediate points (6) shown in Figure 8.7.6 (2). 
The advantage of this procedure is that the (b) points have a maxi- 
mum residual of - 2 when we work to the nearest whole number. 
When the new approximation is complete it is ‘relaxed’ as before 
to reduce all residuals to 2 or less. 

At this point the interval is again halved and the values corrected, 
by relaxation, to have residuals less than or equal to two. The results 
of this process are shown in Figure 8.7.8. 


875, 
750 
625 
500 
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Figure 8.7.9 


For the present purpose this approximation is sufficiently exact, 
but if greater precision were needed the mesh size could again be 
halved and the process repeated. In Figure 8.7.8 a set of equi- 
potentials have been drawn and, as these are frequently required, we 
may remark that they can be located with sufficient accuracy by draw- 
ing a simple graph of the potential values along each net line and 
finding the intersections of this curve with ordinates whose values 
are those of the potentials required. An example of such a graph, 
for the symmetry axis OG, is shown in Figure 8.7.9, In this example, 
where equipotentials intersect OG normally, it is a good plan to find 
the intersections with this line although, in general, such an oblique 
line would not be needed. As a practical point in technique it should 
be noticed that the graph for an oblique line such as OG, can be 
most conveniently drawn by placing the ruled paper with its edge 
along OG and marking the lattice point intersections as shown in 
(a) . . . (a). The contour crossings are also marked on the graph 
at (c) . . . (¢) and can be inserted on the grid by replacing the 


completed graph on OG. 
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8.8. CHECKING 


An advantage of the relaxation method is the fact that errors are 
automatically corrected although, of course, they hold up the con- 
vergence of the process. It is particularly necessary to recompute the 
residuals ab initio periodically since it is rather easy to make mis- 
takes in writing down corrections to adjacent residuals after altering 
any point. 

‘The estimation of the overall accuracy of the solution is more 
difficult, it is not sufficient to assume that when residuals have been 
reduced to -+- 2 units this is the overall accuracy of the solution, since 
no account has been taken of the intrinsic errors due to the finite 


difference approximation. A good guide is available when several. 


grids, of different mesh sizes (8s), have been used in the calculation; 
generally the error of the final solution at any point is less than the 
difference between the values obtained, at that point, on the final 
and penultimate nets. 

An analytic estimate of error can be obtained by computing the 
value of the operator K (8.4.12) at each point, the value —K/20 will 
then give the correction to be applied if K (instead of H) were the 
relaxation operator, since Καὶ is a better approximation to Ὁ " than H 
(8.4.11), this correction is a measure of the error. 

Thus, in our previous example (Figure 8.7.8), the application of K 
at the points a and β leads to values Ra = + 47, Rg = — 38 suggest- 
ing corrections + 2, — 2. Since these are in accord with our previous 
estimates we may take the values given in Figure 8.7.8 as certainly 
reliable to 1 per cent as required. 

Other measures of the accuracy of a solution can be derived. For 
example, if we assume that errors are of two kinds: 

(1) Those due to the approximations involved in solving the 

finite difference equations. 

(2) Those due to approximating the operator \7* by the opera- 

tors H or K, 


then it can be shown (13) that, if the region over which the solution is 
computed can be enclosed in a circle of radius R, then the maximum 
error in the approximate solution of the difference equations is: 


Ea) < |r| R?/e(8s)2 ....(8.8,1}) 


where r,, is the maximum value of the residual at any point, (δε) is 
the mesh size, and ¢ = 4 for the operator H and e = 24 for the 
operator Κα, 
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Errors due to cause (2) are more difficult to estimate, but the 
formula 
ει) S «κί: [N?W | max ....(8,8.2) 
(where N? is the operator defined in equation 8.4.13) gives an order 
of magnitude in the case of operator H whilst for K 


R? 
€(2) < < 91096(85)3 [N*W | sine ....(8,8.3) 
is appropriate. Neither result is valid for boundary points; 8.8.2 
holds, in general, for both the Laplace and the Poisson equations, 
and 8.8.3 for the Laplace equation only. For the special value 
p(x, y) = const., in equation 8.6.12, 8.8.3 may also be used. 


8.9 MONTE CARLO METHODS 


Monte Carlo methods can be derived for the solution of linear partial 
differential equations and have the advantage that, by their aid, the 


ws ” 
p 
V7v20 Lp 
Εν 
x 
Figure 8.9.1 


solution can be obtained at a single point without the need for 
working over the whole field. We give as an example the solution of 
the Laplace equation. 
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Assume that the solution of Ὁ “Κ᾽ = 0 is required at some point P, 
inside a boundary I upon which the values of V are prescribed. 
Starting at P we make a random walk in which the steps, of length 
(5s) and in a positive or negative direction, are made parallel to the 
X or Y axes. The walk continues until it reaches Γ᾿ when it is termin- 
ated, and the value of V, Vp, say, is recorded. This process is repeated 
many times (7) and it may be shown (13) that: 


fi 
Lt 2 Vp |n— Vp. .. (8.9.1) 
n—>oo 1 : 
The method can be extended to more than two dimensions and also to 
the case of the Poisson equation; it seems more appropriate to the de- 
termination of the values of V at a single point than to its evaluation 
over the whole interior of I’, since the number of steps in each walk 
is of the order (R/ds)? where R is the radius of a circle completely en- 
closing Γ΄, and the rate of convergence is proportional only to 1 /4/n. 


8.10 MORE COMPLICATED PARTIAL DIFFERENTIAL EQUATIONS 


The methods outlined in this chapter can be extended, formally, 
to more complicated equations without difficulty. For example the 
equations: 


dw d2w dw 
Ἢ - σία +5) ....(8.10.1) 
dw @? 
Fy = 5,2 (we-™) ene (8.10.2) 
dw ὃ3ωο υ tw 
Ἔ - οί δος +5 a , (8.10.3) 
V2V 2a = 0 see 8 (8.10.4) 
d/ oO ὃ ( ἂψ ἐπε Ἐπ ἐπ δὶ 
κί =| + BAX ay + 2Z(x,9) = 0 [x = x(3)] 
«+ .(8.10.5) 
2 
eet -e-8? x2 0,1>'0 
av 
= BN the = So oP .. ++ (8.10.6) 
τ — 0 x—>o,i>0 
vp = y) x>0,t=0 
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ou du dv dv 
Hx +K(5 + ἀν) thy, +P =0 
dv du 
ἘΣ =0 ᾿ς «ο[8.}0.7} 
H = αΞ — u°, K = — wa, L = α" — υ", P = αὐ 


a® = $(g — 1)(1 — εὐ — νἢ 


appear in the literature and methods of approach can be worked out 
for almost any equation or system of equations which may be sug- 
gested. The difficulty is not in evolving systematic means of iterative 
‘solution’, but of ensuring that the results obtained do, in fact, bear 
some relation to the true answer. 

The reader interested in following up the equations just quoted 
will find them in the references given below: 


8.10.1-3 Muisxe, W. E., ‘Numerical Solution of Differential Equations,’ Wiley, 

8.10.4 New York (1953) ; 

8.10.5 Sournwett, R. V., ‘Relaxation Methods in Theoretical Physics’, Oxford 
(1946) . 

8.10.6 Lanpau, H. G. and Hioks, B. L., Math. Tab. Wash., IIT (1948) 207 

8.10.7 Lorkin, M., ibid., IIT (1948) 209 
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9 
NON-LINEAR ALGEBRAIC EQUATIONS 


9.1 PRELIMINARY IDEAS 


Tue subject of this chapter is the solution of those types of non-linear 
equation, both simple and simultaneous, which do not involve the 
operations of the differential or integral calculus. ‘The problem may 
be expressed as: 

‘Find the solutions of the set of (n) simultaneous equations 


Fe (ig Bes: 90: Spo 9s Ry) ὑπ (ES ooo). ..... 19... 


where the function /; may vary from equation to equation. 
If we take a single function, f say, of a single variable, x, the 
problem is simply: 


‘solve f(x) = 0°. τος ε(8.1.9) 


Clearly equation 9.1.1 embraces 9.1.2 and also the case in which the 
jf, are simple linear combinations of the x;, the latter case leading to 
the simultaneous linear equations of Chapter 7. 

It will be appropriate to consider, too, those cases in which the 
equations 9.1.1. form both under- and over-determined sets, a 
situation which is common in various branches of physical science. 


9.2 GRAPHICAL METHODS 


The solution of non-linear algebraic equations in a single variable 
can generally be best performed in two stages. The first may be 
called a survey in which the general characteristics of the system are 
investigated and in which the roots are located approximately. The 
second stage may be called the refinement, in which the roots are 
found to the desired degree of accuracy. 

It cannot be too strongly urged that even in the simplest problems 
the survey should be carried out, since the feel of the problem, thus 
obtained, will invariably lead to a better use of the available 
resources at the refinement stage. 

Since the advent of high speed digital calculators the technique of 
survey has been slightly altered and, in some cases, the mathematical 
work which had to be carried out as a preliminary to the old methods 
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has been thereby eliminated. Nevertheless the availability of the 
new machines should not be made the excuse for rushing into 
computation rather than contemplating the problem. 

To take a concrete example we may consider the solution of the 
cubic equation: 


ax® +- bx® + cx +d=0. (9.2.1) 
First we notice that this can be reduced to the form: 
x3 + b’x? εἶχα αἰ τὸ ΠΥ ΔΕ 


by division through by a, and for a computing machine such a pre- 
liminary reduction would almost always be wise. If the survey 15 to 
be conducted by hand, however, we should consider the nature of 
the original coefficients a, b, 6, d; should these be simple whole num- 
bers they may be far easier to manipulate than the, possibly, infinite 
decimals of equation 9.2.2. We assume that, in the present case, the 
latter form is appropriate, and consider the next step. On an auto- 
matic machine this might consist of a simple evaluation of 9.2.2 for 
a range of values of x. Considerations of the relative sizes of 6’, c’, d’ 
will limit this to manageable proportions, and may give some clue 
as to the interval of tabulation required. As a minor point of pro- 
cedure it may be noted that 9.2.2 should not be calculated by forming 
(on an automatic computor) successively x°, b'x*, c’x and adding 
these to d’, but by means of the sequence: 


xt, x(x - 37 +e, αἰχία +b) +c] 4d’ 


the reason being that the former process requires four multipli- 
cations to form χϑ, b’x® and c’x, whereas the latter requires only 
two, 

When an automatic machine is not available, however, the above 
method is not to be recommended. A better approach is to notice 
that the roots of 9.2.2 are the same as the x co-ordinates of the 
intersections of 


ΡΈΕΙ 9 }. ....(9.2.3) 
y= χϑ + b’x? | 

The first of these equations represents a straight line, which may be 

conveniently drawn through the two points: (0, — d‘) ( -- d' /c’, 0) 

whilst the second equation is most easily plotted by means of a table 

of squares and a slide rule—the first to find x* and the second to 

multiply this by (x + 6’) which is formed mentally. 
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Figure 9.2.1 (a) (6), illustrates the two methods of approach applied 
to the simple cubic equation: 


x® — 3x2 + 4x —2 = 0. 
Graph (a) is obtained by a direct plot of: 
y = χ — 3x? + 4x -- 2 


jfa) 
yaxt-Jxt4+4x~-2 


ax#(x-3) 
». Ὁ 


[8] 


Figure 9.2.1 


at interval x = 0-5, whilst the pair of graphs (4) give 
ypo—4e42 
J = x(x — 3) 
and were plotted from points at the same interval, both reveal the 
real root at x = ]. 


The above example is an extremely simple one and does not 
show the wealth of detail which can sometimes result from the pre- 
liminary survey; a more interesting illustration is provided by the 
equation: 

ax = tan*x 
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which occurs in the design of gears. We replace the single equation 
by: 


ave He : } 2 -(9.2.5) 
y= 


and thus obtain the graph of Figure 9.2.2. 


Figure 9.2.2. tan x = ax 


The intersections, for various values of a, give the approximate 
positions of the required roots which can be refined by the processes 
to be described in section 9.3. In addition to this data, the graphs 
show that there are no positive real roots (other than x = 0) in the 
range (0, 7) when 0 < a < + 1, and no roots in the range (0, 7/2) 
when a < 0. Furthermore, it is clear that for large x the roots tend 
asymptotically to (n + })z for all values of |a| > 0. 
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An example of a survey applied to a set of simultaneous equations 
is provided by x-ray crystal structure analysis. It became necessary 


γε» 


0 


Figure 9.2.3 (a) 40 [sin (4xx).sin (2ny) — sin (2xx).sin (4ry)] 


to obtain the solution of the simultaneous equations: 


40[sin (47x) .sin (27y) — sin (27x) .sin (4π)}] = — 30 | ore 
40[cos (8πα) -+ cos (8ny)] = + 10 f “7 


and for these the two dimensional contour maps shown in Figure 9.2.3 
(a) (δ) are plotted. By superimposing the two maps, as in Figure 9.2.4 
it is possible to read off the approximate roots of the equations 9.2.6, 
and also to see exactly how the various alternative pairs of (x, y) 
values arise, a point which is by no means clear from a direct 
analytic solution. 

For systems of equations which involve more than two variables 
the graphical survey is generally impossible. In these cases, apart 
from any information as to the locations of roots which may result 
from a knowledge of the experimental system in which they arise, 
there will usually be no alternative but to take some arbitrary 
starting point and use one of the successive refinement methods to be 
described in the remainder of this chapter. A word of warning is 
perhaps appropriate in this connection; it is that refinement methods 
usually postulate that the trial point is near to the true solution. If 
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this condition is not satisfied, they may either not converge at all 
or, more probably, converge to a false solution which is in the nature 


+80 ἐκ | +80 
X= "5 


Figwe 9.2.3 (δ) 40 [cos (8x) + cos (8ry)] 


of a ‘col’, or depression, in a mountain range, instead of the true 
valley which is sought. - 


γε 


NON-LINEAR ALGEBRAIC EQUATIONS 


9.3 ITERATIVE PROCESSES—ONE VARIABLE 


We may, as in section 9.1, write the general non-linear equation in 
one variable as: 
T(x) = ἢ sae . (9.3.1) 


and the problem which we shall consider in this section is the solu- 
tion of this form. We may remark that when /(x) is a polynomial, 
say: 


Six) = x" + ax?! + age? 2. ta, = 0 .... (9.3.2) 


classical methods of solution, depending upon the theory of equa- 
tions, are available. We shall not mention them further except to 
remark that they are the methods of Horner and Graeffe-® and 
that they find little favour with modern numerical analysts. The 
analytic solutions of the cubic and diquadratic equations, known as 
Ferrari’s, Tartaglia’s and Cardan’s methods, are of little utility in 
numerical work. 

An iterative process for the solution of an equation of the type 
9.3.1 may be defined by the statement that, if x, is an approximation 
to the solution of equation 9.3.1, it enables a quantity x, to be 
calculated by means of some relation: 


x, = I(x») ee {9.9.9} 


in such a manner that x, is a closer approximation to the required 
solution than was xp. 

If x) differs from the true root by a small quantity, of order (e), 
say, then the iterative process 9.3.3 is said to be nth order if the error 
in x, is of order (e"). Most of the best iterative processes are second 
order; third and higher order processes exist and can always be 
constructed from those of lower order, but often they involve greater 
total computing labour for a given final accuracy than those of the 
second order. 


As an example of a first order process we may consider the so- 
called regula falsi or rule of false position which is possibly the oldest 
known iterative procedure. 

Suppose it is required to find the root of: 


y = flr) =0 

corresponding to the intersection R shown in Figure 9.3.1. Any point 
(Xo, %)) on the curve is taken and also any other point (x,, y,)—pre- 
ferably on the opposite side of OX from (x, yp). Effectively the process 
consists of finding the intersection of the line joining (x9, 97) (*1; 91) 
with OX and then taking this abscissa, x, say, to determine a new 
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point (x,, v2) on the curve. The operation sequence is repeated using 
(xq, 99) and (xq, yg), and it is clear that convergence to the root, R say, 
takes place. Analytically the method is equivalent to a sequence of 
linear interpolations, and the relation between successive convergents 
is easily shown to be: 


Xnt1 = (XoIn — XnIo) (In — Io): ve +(9.3.4) 
To investigate the relation between the errors of the nth and 
(n + 1)th approximation assume the root to be x = X and let: 
Χρ = X + €& 
Xn ΞΕ X + «, 


(x), γι} 


Figure 9.3.1 


Then Taylor’s theorem gives: 
70 = of '(X) + beof"(X) +. --- 
In = Enf'(X) + den (Δ) + ++: - 


(since, by definition /(X) = 0) and substitution in 9.3.4 and simplifi- 
cation leads to: 


Xnty =X + heo-enf (XD (X) 


whence | 
ἐμ: = b€-€nf (Δ) { (Δ) ....(9.3.5) 
thus, if εὐ is such that |}e,f"(X)/f'(X)| <1, the process will 
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converge. Actually convergence may be quite rapid if the initial 
point is well chosen, but it should be noticed that each iteration 
of the process will only produce an improvement of the order of 
one place of decimals. 


Another well-known technique for solving polynomial equations 
in one variable is the method of iteration. The equation to be solved, 
f(x) = 0, is written in the form 
x = g(x) 
The iterative process is then simply : 


Xngr = 8(%n) 

Clearly, ifthe process converges #! x, tends to the solution required. 
Ἡ -ΟΣἹ 
To investigate the convergence of the process assume that the 
approximation, x, differs from the desired root a by a small quantity 
e. Then, since a is a root of f(x) = 0, 
a = g(a) 
Also Xn41 = (Xn) = g(a + €) 
Whence, expanding, by Taylor’s theorem, and neglecting terms in 
εἶ elc., 
Xn4¢1 = g(a) + eg’ (a) 

=a + €g'(a) 

It follows that the process converges if |g’(a)| < 1. 
The advantage of the iterative method is its simplicity, its disad- 

vantage is that, since a is initially unknown, it must often happen 
that convergence cannot be determined a priori. 


A general technique of construction for second order processes is 
that known as the Newton-Raphson method. Assume that an ap- 
proximation (x,) has been calculated which has an error (e€,), then 
since: 

ΤΑ + &) =9 


we have, by Taylor’s theorem: 
| 
flen + εὐ = 0 το γα) + enfin) + GS") + 
so that, if we choose 


Nay = Kq — Stn) Uf a) ....(0.3,6) 
the error in x, 4, is of order (€,)*. 
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It should be noticed that there is no unique iteration of the second 
order leading to a given result. Thus, suppose that a process for 
determining the square root of a quantity ὁ is required, the equation: 

χΞ -- ὃ τεῦ a's oe) 
has the required solution x = + +/d. 

From equation 9.3.6 the iteration: 

Sata = tn — (ἃ — b) /2x, = 4(%n + b/x,) ....(9.3.8) 
is obtained. On the other hand, 9.3.7 may be written: 


L/x? —1/6 =0 
and this leads to: 


Sua = Xn — (ἢ — 1/b)/(— 2/28) = xaLl + (6 — 22) /28).- 
εὐ. ο[9.3.5) 

It can be shown that, although equations 9.3.8, and 9.3.9 are 
both second order processes, equation 9.3.8 converges rather more 
rapidly since the coefficient of εἰ is less. On the other hand, 9.3.9 
requires only one division (to form 1/26) and this may be an advan- 
tage if it is to be used on an automatic digital calculator which, as is 
often the case, has no automatic divider. 

Simple rearrangement of the original equation is not the only 
method for obtaining second order processes. Thus we may multiply 
the original equation /(x) = 0 by an arbitrary function g(x) and 
apply the Newton-Raphson process to the combined system. If we 
take g(x) = x and apply this to equation 9.3.7 there results: 

x — bx =0 


for which the iterative solution formula 15: 


δ᾽ τὺ 
Kosa πὰ ioe ἘΞ — = 2x3/(3x2 — δ). .... (9.3.10) 


Now although both equations 9.3.8 and 9.3.10 are second order 
processes it is easy to show that, if: 
xy =vV/b+e 
equation 9.3.8 leads to: 
4n+1 = a/b a ξεν ὃ πα 1ε3} oa . εν 
and equation 9.3.10 to: 
Xn4y = Vb + ξεῖν —F/b+... 
Whence, by taking the iteration defined by: 
8(9,3.8) — $(9.3.10) 
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the term in εξ ν΄ is eliminated and we obtain the third order 

process: (8) 

Xna1 = (Sed + 6bx5 — 36%) /4x, (3x5 — δ) + e8/b+... «... (9.3.11) 
Similarly from: #(9.3.8) + 2(9.3.9) 

we obtain the third order process: 

Xna, = (3b? + Θὐχ — xp) /8bx, — Ze*/b +... «...(9,3.12) 
and 9.3.11 and 9.3.12 could be combined to give a fourth order 
process. It is quite apparent from the complexity of 9.3.11 and 
9.3.12 that they are of little utility in practical computation. 

We may notice, before leaving the subject of second order pro- 
cesses for simple functions, the formula: 
ρα = Xn(2 — bxp) -+++(9.3.13) 
which converges to | [ὃ and is derived from the equation: 
l/x —b τεῦ; 
and also the iteration: 
ρα = χα + 1) — ban] /p »+++(9.3.14) 
which converges to 1/5? and is derived from: 
1/x?—b=0. 


9.4 COMPLEX ROOTS 


The Newton—Raphson process, described above, is usually considered 
to be applicable only to the determination of real roots, but actually 
it is quite feasible to use the Newton—Raphson process for evaluating 
complex roots, and since this is applicable to quite general equations 
and is second order, it will now be described. 

Assume that it is desired to find a solution z = a + ib to the 
equation: 


f(z) =0 τ (9.4.1) 


Let z= % + ἢ, where ἢ = ε + ἰδ, and ε and 6 are assumed to 
be of the first order of small quantities. Then Taylor’s theorem 
gives: 

S(2) = 0 = f(%o + η) το ἴσο) + af'(%0) + ἐη (τ) + . - - 
whence, to the second order: 
ἢ = — f (2%) If (2) eens (9.4.2) 
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which is identical with the ordinary Newton-Raphson formula for 
real roots. We now note that: 


ε = real part of [— (29) {{᾿{29}} 
ὃ = imaginary part of [— f(#») /f’(»)] 
and that, to start the iteration, it is necessary to take a complex 
value of (29) which in the absence of other information can be 
taken to be (i). 
As an example of the convergence of this process the following 
figures, obtained in solving: 


z?@—2z:+2=0 
from the initial value z, = i, show it in a favourable light: 
% =i No = (3 — i) 
% = (1 +i) mh = ge(13 + 9i) 
Ze = 25(43 + 39i) ἢᾳ = — .07673 + -02230i 


Zs = .99827 +- -997301 
The true solution is evidently (1 + i) and the errors; |] + i — z,| 
are 1, «35, "08 and -0032 respectively. 
On the other hand, if an attempt is made to find directly a com- 
plex root of the equation: 


22 — 328 +42 -—-2=0 


discussed in section 9.2, starting from the same initial approxima- 
tion (i) the successive convergents are: 


zy =i no = "46 — +243 
z, = "46 + -76i n= "43 — «1921 
χε = "85 + «64 ny = "825 + -98ὶ 


24 = 1-54 + -78ὶ 


with errors 1, «59, +39 and -58, so that z, is a less good approxima- 
tion than z,. The reason for this behaviour lies in the fact that the 
derivative of 


χ8 — 322 -+42-—2=0 
that is: 323 — 6z + 4 
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has a pair of complex roots | + +577i and that the value 
ze = "85 + 641 
is fairly close to one of these. 


The methods for determining complex roots of polynomial equa- 
tions, which are usually recommended, are based upon the deter- 
mination of the real quadratic factors of the polynomial. ‘This avoids 
the need for complex arithmetic because the complex roots occur in 
conjugate pairs and, hence, imply a quadratic product with real 
coefficients. 

Three methods for determining such real quadratic factors are 
in vogue: the method of iteration, usually known as Lin’s method, 
the Newton—Raphson method, and Bairstow’s process. Lin’s method 
is first order and is based upon the following consideration. Suppose 
that our polynomial 


ΤῸ = ω baz +...+4a=0 = ....(9.43) 


is assumed to have a pair of complex, conjugate, roots a and @ 
which satisfy the quadratic equation: 


4(2) = ζῇ +6,2+6,=0 εν ν (9.4.4) 
Then 
b=-—a-@ es 9.8) 
and 
be ΞΞ- aa ee (9.4.6) 


so that 6, and 6, are real. 
If the quadratic 9.4.4 is an exact factor of 9.4.3 the result of 
dividing f,(z) by q(z) will be: 
In(Z) = Sn—2(2) -9(2) 
where /,_o{z) is some polynomial, of degree (n — 2) 
faces) τ ag ts tee esx (94-7) 


In general g(z) will be unknown; we therefore assume some trial 
quadratic φοί 2) = z? + b,9z + bo) and attempt to divide f,(z) by it. 
The result can generally be written 


fn(Z) = 8n—2(Z)9o(Z) + p(z* + by,2 + 691) 
where g,,_»(z) is a polynomial of degree (n — 2) in z, of the type 
gn—2(z) = ἄρον 
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In Lin’s method the remainder polynomial g,(z) = 2* + 6,z + bg 
is used as a trial divisor and the process is repeated to obtain a 
sequence of quadratic divisors g;(z) = σῇ + δγιΖ + bg; until, if ἢ 
converges, 9; 4.1 —> q at which point q,(z) is clearly a quadratic divisor 
of f,(z). 

Woes a fairly good guess for the initial trial is available, Lin’s 
method generally converges quite well, but convergence can be slow 
or non-existent and, for this reason, one of the methods now to 
be described is usually preferred. 

The Newton-Raphson method for quadratic factors is second 
order, to derive it we assume that we are at the ith stage of an 
iterative process and that we have an approximate quadratic factor: 


gi(z) = ζῇ + bz + do; ... + (9.4.8) 

corresponding to 9.4.5 and 9.4.6 we have 
b, = — αἱ — a, soe + (949) 
bo; = aja; «+ (9.4.10) 


Suppose that the true roots of 9.4.3 are (a;+ 5a;), (αι + δαὶ). 
The Newton—Raphson process, applied to 9.4.3, gives: 


δα; = — fn(ai)fn(ai) εν. (9.4.11) 
δᾶ, = — faldi) π(αὴ 1. « (9.4.12) 
whence, from 9.4.9 and 9.4.10, the changes réquired in 6); and bo; 


are given by: | 
δι = - δα; = da; sees (9.4.13) 


δύω, = ada; + ada, .. + (9.4.14) 


To determine the various quantities involved we proceed as 
follows. Divide f,,(z) by φιί Ζ) to obtain 


Sulz) = 92) fn-2o(2Z) +nz+5 .... (9.4.15) 

from which, since a; and G; are the roots of ¢,(z) = 9, 
Snr(ai) = παι + Sy ... (9.4.16) 
Sa(G) = HG + “ἃ 2. (9.4.17) 


Next divide f;,(z) by 9¢;(z), obtaining: 
ΚΖ) = 9(Z)8n—2(Z) + 19z + 59 
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whence, as before, 


Fn(ai) = rea; + 5p εν ν . (9.4.18) 
Ln(Gi) = σρᾶ; + 50 ....(9.4.19) 
Substituting for 9.4.18-9.4.19 in 9.4.11 and 9.4.12 we get: 

i ag ns: OE 5Ὲ αἱ 
ὕχαι + Se 
is ra; + S$ 
δ eo Li 
‘i TG; + Sp 


and, substituting these values in 9.4.13 and 9.4.14, and using 9.4.9 
and 9.4.10 to eliminate a; and G;, there results: 


_ ὥγγρδοι — (152 + Te5,)b1i + 25459 ΒῚ 
8b); —NT —_ ribo; 7 ToSabyi +3 ἃ ἃ ἃ (9.4.20) 
and 


Sho; -- (abbas — 2γμεῦει + Sa5abre — τειν, — 969) 
a γῆδοι — TeSqbii + 53 
εν ν (89.4.21) 


We now define: 


Gin i(Z) = ζῇ + (by; + δέχ) + (bog +°5bg¢) = ζῇ + ὀγεειΖ + boi41 


and repeat the process on q;4;(2Z). 

The various quantities r,, 5), 7.» 5g are formed by the usual process 
of synthetic division. 

Bairstow’s process is as follows. Assume, as before, that an :th stage 
approximation q;(z), defined by 9.4.8, is available. We again form 
the expression 9.4.15 obtaining f(z), ἢ and s,. fy—o(z) is now 
divided by the trial quadratic giving: 


Sn—2(Z) = QilZ)fn—4(Z) + τῷ + 50 
The corrections 5b,; and 8b.; can then be shown™? to be: 
14S — ToS; 
ne δ 2 
δ Hibey το στρογ + ἢ 
γεν — Τρῖχῦν σὴς 
γῆϑοι — TeSebyi + 53 


A comparison of these equations with 9.4.20 and 9.4.21 shows that, 
when /,(z) is a quadratic, Bairstow’s process gives an exact result 
whereas the Newton—Raphson method does not. 


8b; = 
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9.5 EQUAL ROOTS 


When an equation is such that it is satisfied for values of the variable 
which tend to equality, with some parameter in the equation, the 
Newton-Raphson process will, in general, break down. Geometric- 
ally the situation is simply that the curve: 


) =f(*) 


is nearly parallel to the x axis in the region of the roots (or, in the 
limit touches the x axis), and this should have become evident in the 
preliminary survey. 

When exactly equal roots occur they can be found by the simple 
process of extracting the highest common factor of f(x) and /’(x). 
When exact equality is not present, however, the first step should be 
to find the root of f’(x) = 0, say (χη). Then assume that the roots of 
f(x) = 0 are (%m + €) so that: 


S(%m + ε) =0 = χω) + ef (Xm) + de?" (Xm) ee 
or, since by definition f'(*,) = 0, 
e= = fm), PS be 
+ γ Fa) (9.5.1) 


From this point it should be possible to revert to the normal 
Newton-Raphson process with some hope of convergence, 

Alternatively, when a pair of equal roots is known to exist, an 
approximating value, x, may be improved as follows. Assume that 
the correct solution is (xy + ¢), then Taylor’s theorem gives: 


ας. ὁ =S(%o) + εὐ) + ἐε ὦ Ἔ - - 
Since a multiple root occurs at (x) + εἾ this must correspond to 
the minimum of f(x, + εἰ with respect to ε, 


Yom f'(n) ἘΦ 
whence: e = —f' (x) | f" (*0)- ...«.(0,5.2) 


9.6 ERRORS AND INACCURACIES 


The solution of polynomial equations of high degree is, nowadays, 
almost always conducted on an automatic digital computer. The 
precision with which the calculations are to be conducted needs 
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careful examination and the following treatment, due to J. H. 
Wilkinson, is instructive. 
Suppose that the polynomial equation is: 
f(z) Sz" +az"*+...+4a,=0 


and that we wish to examine the sensitivity of any root, say a, with 
respect to variations in one of the coefficients a,. Let a, become 
(a, + δα.) and assume that the new solution is (a + da). We then 
have 


f(a + 8a) + δα (α + δα)"-" = 0 
Or, neglecting terms in 6°, 
f(a) + δα (α) + a”~"da, = 0 

But f(a) = Ὁ whence: 

Ba ner Oar nr 8; 8, 

a f'(a) f(a) a, 
Wilkinson points out that if nm = 20, and the quite well behaved 
polynomial f(z) = (z + 1) (z+ 2). ..(z+ 20) is considered, 


then for a = — 16 the root is particularly sensitive to variations in 
the term a,. Thus 

δα -- — tow x 210 δα; ~~ 3-2 χὶ 1010 δα; 

a 4! x 15! ay ay 


so that calculations must be carried out to better than ten place 
accuracy. 

The analysis points up the fact that, for equal roots (where 
f'(a) = 0), the solution to any equation is extremely sensitive to 
variations in any coefficient. 


9.7 SIMULTANEOUS NON-LINEAR EQUATIONS 


We return now to the solution of the general set of non-linear 
simultaneous equations envisaged in section 9.1 


Kili, ἴα... eee fee)... "πὶ Le, 


that is, a set of m equations in n unknowns. When m = n we have the 
classical case in which a finite multiplicity of solutions may exist. 
The more general situations, in which m <n or m> 7n are also en- 
countered in physical problems and, in these circumstances there 
may be no unique solution in the former event, and no solution in the 
latter. 
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To unify the method of approach we shall define a solution of 
equations 9.7.1 to be any set of parameters (x;) which makes the 
positive definite form: 


= 5Gi(f) (9.7.2) 
i=] 


a minimum, G;,(z) being a function so chosen as to be always 
positive for all values of z, real or complex. 
For convenience and simplicity it is usual to take either: 


G,(z) = Χ. 
or G(z) = |2| 


where Z is understood to mean the complex conjugate of z. 


The general approach to the solution of a set of equations of 
type 9.7.1 should again be in two parts, first a survey to detect the 
rough location of the roots, and second a refinement process to 
evaluate the roots more exactly. Unfortunately the survey process is 
almost impossible when more than two variables are involved and 
in this event a refinement by successive approximation may have 
to be attempted. 

When only a small number of equations and variables are involved 
(up to three) it may be feasible to use the Newton-Raphson process 
to find a solution to the required accuracy. Thus suppose that the 
required solution is at (x, + €&, %2 + €:- + -+%n-+ €n), then the 
n variable form of Taylor’s theorem gives: 


Fil + ey Xa + €n e+ + Hn + ἐμ) 
= fi(xy %_ . » « Xn) + Ee, + O(e?) (Gat... δ) 


whence, for a second order process, we take (εἰ . . . €,) to be the 
solution of the set of linear simultaneous equations: 


Of; 0 Of; , 
«ἀν Ὁ αδδννν Ἔα δὲ τι τῇ ἐξ πεῖ...) ces FS) 


aay n 
which correspond exactly to the one-dimensional Newton-Raphson 
process of equation 9.3.6. 

When large numbers of variables and equations are involved this 
process will, in general, be too complicated to apply, and under 
these circumstances the analogue of one of the successive approxima- 
tion methods of Chapter 7, sections 7.7. and 7.9 is to be preferred. 
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If the function P(x,, x... .ἕ Χρ), defined in equation 9.7.2, is 
taken as the basis of a minimization process, the analysis can be con- 
ducted in a manner which closely parallels that of sections 7.7 and 
7.9 with the exception that the n dimensional surfaces: 

Φ = const. 

are no longer, in general, hyper-ellipsoids. 

To take a two-dimensional example, the contour map of Figure 
9.7.1 might be encountered. In this two maxima are shown at M, 


Figure 9.7.1 


and M,, a minimum at m and a ‘col’, or pass between mountains, 
at C. 


We seek to progress from an initial point of vector & (= x,x_.. . Xp») 
to a vector (& - aw) which makes ®(€ +- aw) less than @(€). The 
Taylor expansion gives, at once, 


n δῷ n a2 | 
P(E + av) = P(E) + kid au, ax. + =. ξαῦυ,υ, Dens + O(a*). 
τς εἴ8,2.4) 
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Here v is any vector and », is its component in direction x,. Assume 
now that it is adequate to represent the variation of ® along the 
vector v by meansof the terms of degree less than three ina in equation 
9.7.4. We then obtain for the least value of ® the multiplier a by 
means of: 


d® 
Ἔ Ὁ 0, 
That is, from equation 9.7.4, 
ὃ] = δ᾽ Φ 
= - ---ἰ 2 υ,.Ὁ, Ξ--Ξ:-:. ΟΠ ὦν 
on λυ, δικός 9195 


Just as in Chapter 7, there are several rational choices for the 
vector v. If it is taken parallel to one of the axes of co-ordinates, x, 
say, equation 9.7.5 becomes: 


ad [ὃ: Φ 


~ ὄχι! Ox,” τὶ αὐ θ τὴ 


ἄ ΞΞ 
and this corresponds ἰο ἴῃς relaxation technique of the linear case. 
For a steepest descent, v is defined by: 
n  @@ 
=grad@®@= 2 ὁ, — - 9 πὶ 
: gr r=] Ox, 
where e, represents the unit vector parallel to the axis of 4,. Substi- 
tuting in 9.7.5 we obtain: 


Σ(Ξ " δῷ oD aD 


ΡΥ ὃχ, rom Ox, j Ox, ᾿δχ,ῦχ, 


«+ (9.7.8) 
r= 1 
so that the actual change in any co-ordinate x, is: 
we Σ (sr) | 3 ὃ OD OD (9.7.9) 
4 Ox, OX pus \OX, panitOXp OX, OX,.O%, 

This formula is really too complex for use in actual computation 
and although it has been applied, is not so simple as a variant 
which will now be described. 

Let the value of ® along v be as shown in Figure 9.7.2. 


We first take the intersection of the tangent to the ®, a curve with 
the line ® — 0, Tsay. OT is simply determined by means of equation 
9.7.4 by neglecting terms in a* and so on. ‘Thus: 
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n a 
opm — O(6) / Ea 5 (9.7.10) 


Next we calculate the values of ® at the points (E +a;v) and 
(ξ + 4a7v) and call these D,, and ὦ, respectively. 


Figure 9.7.2 


| 

| 

ιν | 

| 

ῦ GB 


a 


The minimum is then calculated by the simple process of forming 
the quadratic approximation to ® which passes through ®, ®, and 
®,, and then differentiating it with respect to a. Thus: 


whence, since oy = 0, 


da 


a 
α, Ξ (6 Φ — 24Φ)44Φ. ex (WIAD 


Thus the individual component changes, av,, are for the steepest 
descent, simply: 
(ὦ, -- 45, 34) 0.90), 
ἀ(Φ, -- 20, + Φὴ ᾿ [Σ(5}} τοὺς ah devon) 


r=l1 Ox, 
It will be noticed that the use of this version of the steepest descent 
process requires the calculation of three values of ®, and all = 
Xp 
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2 
(n + 3) values in all, as compared with about 3 values of 
ap ap 


dx, ni Ox, 0x, 
for the analytic expression 9.7.9, 

The method of minimizing ® with respect to variations along a 
single vector v can be extended to cover the case of two vectors 
v and εὖ. This requires the minimization of Φίξ + av + Bw) with 
respect toa and and follows the same lines as those used in deriving 
equation 9.7.5. Just as in Chapter 7, section 7.7 the vectors v and w 
are taken to be the current direction of steepest descent and the 
direction of the last descent; the formulae are, however, so complex 
as to be practically useless. 


To conclude this section we may mention some of the difficulties 
which accompany the solution of non-linear algebraic equations by 
minimization methods. These stem from the fact that the surfaces 
Φ = const. are not simple hyper-ellipsoids having a single common 
centre corresponding to the minimum of ®. As shown in Figure 9.7.1 
maxima may occur, as at M, and M,, relative minima are possible, 
and ‘cols’ are of common occurrence. Each singularity of this type 
is a possible stationary value of ® and care must be taken to ensure 
that any ‘solution’ obtained by descent methods is a true minimum 
and not one of these spurious addenda. This is particularly true in 
the case of redundant sets of equations where an actual zero of @ is 
not to be hoped for. 

It may be noted that, whilst maxima and ‘cols’ are easily detected 
by virtue of the fact that for the regions near to these the quadratic 
form: 


δ 
Μ Ox,Ox, ὦ ze 


is not positive definite, a relative minimum is quite indistinguishable 
by any purely mathematical test from the true minimum which 
defines the desired solution. 
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10 
APPROXIMATING FUNCTIONS 


10.1 DEVELOPMENT IN SERIES 


Tue advent of automatic digital calculators has resulted in the 
frequent necessity for means of generating various common functions 
using only the operations of elementary arithmetic. Thus it is useless, 
from the computing machine viewpoint, to say ‘calculate sin x’ 
since the machine will not normally understand a direct command 
of this type. 


In practice a statement such as that made in the previous para- 
graph could be replaced by the equivalent: 


3 5 7 
Form (--ῷ a 


taking sufficient terms so that the result is correct to the number 
of places required. In order to minimize the amount of work re- 
quired to calculate the value of a given function it is thus appro- 
priate to consider whether a better approximation than the Taylor 
series can be found for a given amount of work. 


To make this more precise we may consider the following proposi- 
tion: determine the (N +- ) coefficients a, so that the maximum 


difference between f(x) and z ae I is as small as possible for a given 
range of values of x. Or again; determine the values of a, so that the 
difference between f(x) and Σ a,x" has the least possible mean square 
value for a given range of ἔν 


Which of these definitions will be best in any given context depends 
essentially upon the ultimate use to which the approximations to 
J(x) are to be put. Clearly if only a single value is required the maxi- 
mum divergence should be made as small as possible, on the other 
hand, if a number of such values are to be associated, either directly 
as: 


M 
2 f%) ἐφ. [10.1.1 
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or with a weighting function w(x,) 
M 
2 w( xs) f(%s) κὸς Ἐ{ 1}. 1..} 
sm] 


then some approximation of least integral square deviation will be 
best. 

In this chapter we shall consider several useful approximations of 
both categories. 


10.2 LEAST SQUARE APPROXIMATIONS 


Consider any set of functions O,(x) (n =0 .. . 00) ortho-normal 
in the interval (a, δ). We shall show that if an arbitrary function, 
fix), is expanded in a Fourier series of these functions, then the 
first (n -+- 1) terms of that series constitute the best (n + 1) term 
approximation to f(x) in the least square sense. 


For suppose that the expansion is: 
ὩΣ 4,0,(x) .. (10.2.1) 
r=( 
then it is required that: 
ὃ n 2 
L -| Ke = 5 0,0,(%) | Be ra δα 
a ima 
is a minimum. Now for a minimum: 
a ae or eee (10.2.3) 
da, 
whence: 
b ‘ , 
| O,(x) | Ae) — § 0,0,(x) |as =0 (s=O. n) 
re 
: ..(10.2.4) 
but, since the functions O,(x) are ortho-normal in (a, δ) 
b 
| OA#)Og(x)Ax τα Bye (....10.2.5) 
Jes 
where §,, is the Kronecker delta function defined by: 
= O(r + s) 
fe —l(r=s) 
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whence equation 10.2.4 may be written: 


ὃ n 
| Os(a)f(x)dx = Σ aBre = α, .. «+ (10.2.6) 


but this is simply the definition of the coefficient of O,(x) in the 
Fourier expansion of f(x). 

The approximation can be generalized to include a weight 
function w(x). Thus the functions O,,(x) satisfy: 


b 
| O,(x) O4(x)w(x)dx = Sy, .... (10.2.7) 


and : 
Ly = | w(x) | Ae) — £4,0,(4) | ‘de .. ++ (10.2.8) 


is a minimum when 
b 
a= | w(x) O,(x) f (x)dx ...+(10.2.9) 


which is, again, the normal Fourier coefficient. 


10.3 SOME USEFUL FUNCTIONS FOR LEAST SQUARE 
APPROXIMATION 


From the classical viewpoint the most important set of orthogonal 
functions are sin (nx) and cos (nx), which lead to the well-known 
Fourier expansions in (0,7) and (0, 27). These will not be further 
considered here save to remark that, under suitable circumstances, 
not only is the mean square difference between function and ap- 
proximation minimized, but that the same is true of derivatives. 

The most important polynomial expansions, for unit weight 
function, are these in terms of the Legendre polynomials P,,(x). 
These are defined by: 


P(x) = 5a ἄπ a (x? — 1)” το i 
Pyisa(x) = [(2n + 1)xPq(x) — mPya(x)] (α +1)... «+. (10.8.2) 
with orthogonality relations: 
[ Pa(s)Pa(s) Ξ- 2δ,,κἰ(2π - 1). .....(10.3.3) 
-} 
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For intervals other than (— 1, + 1) it is easy to transform the 
Legendre polynomials by means of: 

2 b+a 


δι ΘΠ τα 


which gives a set of polynomials Q,(z), say, orthogonal in (a, δ) and 
with orthogonality relations: 


i = 


(10.3.4) 


b 
[ Qal2)Qm(2)dz = ὁ — o)Bmq/(2n +1)... (10.3.5) 


The first eleven Legendre polynomials for interval (— 1, + 1) 
are given in Table 10.3.1. 


Table 10.8.1. Legendre polynomials for interoal (— 1, + 1) 


ny Ῥ,(), fi P(x) 

01 6 | με (231χ — 315x* + 105x? -- 5) 

l x 7 | gg (429x? — 693x5 + 315χ3 — 35x) 

2 | | $(3x* — 1) | 8 τὐκί(δ435χ3 — 12012x* + 6930x4 — 1260x* + 35) 

Ἶ ἐ(5χ — 3χ) 9) 9, (12155x" — 25740χ + 18018x* --- 4620x* + 315x) 
4 4(35x* — 30x7+ 3) | 10) =4,(46189x° — 109395x* + 90090x* 

5 ἐ(63χ" — 70x* + 15x) — 30030x* + 3465x? — 63) 


Useful expansions are: 


ὡς τ ΤῊ! .P,(x) + 5.5 P(x) 


2n(2n — 2) 
+9. (Qn + 3)(2n + 5) -Py(x) +. . if 
Δ ΣΝ περὶ 
2n(2n — 2) 
ἘΝ πες +7) ΡΠ. 1 


(1 -- 28h = 5] ψριρὺ — 5.404)" Pas) 
1.5 


—9.3 (57 53). P,(x) — 13. (sag) Pot we" J] 
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( -- κῆτος 5 Pal) +5. (4) Py) 


+9.(5°5) Pals) +18. (573) Pals) +.» | 


sin-M(x) =F 8-Pi(s) +7.(8)*Pa(s) +11. (7-3) Pals) +... | 
sin (5 r) = 3.3,(7/2)Py(x) — 7.0; (2/2) P3(x) 
+ I .Juy(m/2)Pp(x)—-  « 


cos (5 x) = 1.J3,(m/2)Po(x) — 5.J (1/2) P.(x) 
+ 9.J3(7/2)P,(x) — 
where J,(x) is the vth order Bessel function. 
Polynomials appropriate to (0,00) and for weight function e—* are 
those of Laguerre, they are given by: 
L(x) = @ ὦν νν νιν (10.3.6) 
dx” 


Lnaa(*) = (22 + 1 — x)Ly(x) — n*Ly_1(x) .... (10.3.7) 
and satisfy: 


[ διρλβωροττα =(n!)*Sinn .---.(10.3.8) 
ῦ 


Finally, for interval (— οὐ, + 00), there are Hermite polynomials, 
which are defined by: 


H, (x) = (— 1)" = (e-*") ... (10.3.9) 


Ay +(x) = 2xH,(x) — 2nHy_,(x) .....(10.3.10) 
and have orthogonality relations: 


[ H,(x) Hn (α)ο ἀκ τα 35. πἸνϑπ δι... ....(10.3.11) 
-ο 
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The expressions for some of the above polynomials are given in 
Table 10.3.2. 


Table 10.3.2. —_—_evev 


n | popes. rented’ ale. Hy(x) 
ora 1 

1 | —*+1 2x 

2 x? — 4¥+2 4x? 2 

3 — x? + 9x? — 18x +- 6 8x3 — 12x 

4 | x* — 16x3 + 72x? — 96x +- 24 ΕΓ Ἢ 
5 


-- 85 Ἔ ἀν" — 2000" Ὁ Οὐ» — 600x + 120 32x — 160x* + 120x 


10.4 LEAST ABSOLUTE DEVIATION APPROXIMATION 


A remarkable set of orthogonal polynomials is that derived by 
Chebyshev. These are defined by: 


Ts =1, Ty (x) = smi 608 (ncos“4x) (n>1) ....(10.4.1) 


ΤᾺ χ(χ) = xT Q (x) — ΣΤ΄ γ() ...+(10.4.2) 
and have orthogonality relations: 
+1 | dx " 
* * '. 
ᾧ ΤΣΩΤΑβῸ [A = χίξεδων ....(104.3) 


Their importance lies in the fact that, of all polynomials of degree 
n, T*(x) has the least maximum value in (— 1, + 1) when the 
coefficient of x” is taken to be unity. For consider the function defined 
by equation 10.4.1, it is easily seen that the maxima and minima 
occur at: 

xp = cos ἀπ|π 


where & = 0,1 .. . n, and that for these points: 


TH (xp) = (— 18/2", 
Let 
M,,(x) = x" + ax") + age"? 4+ 1 ww ty 


be a function, not everywhere zero, whose deviation from zero in 
(— 1, + 1) is, if possible, less than that of 7*(x). Then: 


ΤΑ (Xo) — My(Xo) > 0, TH (xy) — My(xy) <0, TH (x2) — Μ,(χῳ) > 0 ete. 
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that is, the function 7;*(x) — M,(x) has roots between (xp, *) 
(x, Χ4) . . « (X,~1) ¥,). There are n of these, and since 77 — M, is of 
degree (n — 1) at most, we are led to a contradiction, it follows 
that the assumption that M,(x) is different from zero and Tj (x) 
is false. 
Modern usage“) favours the slightly modified definition: 
T,=1 T,(x) = cos (n cos x) ..+- (10.4.4) 


T+ = 2ΧΤ, (x) — Τὶ . +» « (10.4.5) 
5 dx π 
| Τα Tals) FH Ὁ Ἢ (MANY 10.4.6) 


= ΠΤ (m =n = 0) 


which has the advantage of avoiding fractions in the values of 
T,,(x), the first eleven of which are given in Table 10.4.1. 


Table 10.4.1. Chebyshev polynomials 


π᾿ Tn(x) n Tn(x) 

δ, 6 | 32x¢ — 48x* + 18x? — 1 

GIs 7 | Gta? — 112e8 +. 56x98 — 7x 

2 | a2—1 8 | 128x¢ — 256x* + 160x* — 32x? + 1 

3 | ἄρας, Be 9 | 256%? — 576x? + 432x§ — 120x9 + 9x 

2 | pees | 10 | 512. — 1280x4 + 1120x* — 400x¢ + 50x? — 1 
5 | 16x — 20χϑ + 5x 


Useful expansions are: 


͵ ἐν ΑΝ 
bled i (ἢ) "παν 
where [4n] represents the largest integer in (4n) and the coefficient 
of T,(x) (if present) is halved, 


7 


aed 7 
and: sin(5 x)=2 Σ (-- γυν α(5) Toa) 


π 7 2 3 7 
cos( 5 , Ξ- Jo(5) +2 =. (— 1)'Je(5] ἘΚ 
where J,(x) is the rth order Bessel function. 
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These Bessel functions, and also those required in the Legendre 
Polynomial expansions, do not appear to be readily available; we 


give their values in Table 10.4.2. 


Table 10.4.2. Bessel functions for Legendre and Chebyshev interpolation 


Jn (m/2) I(n + ἐ)(σ|2) 
0-47200 12157 7 0-63661 97723 7 
0-56682 40889 | 0-40528 47345 7 
0-24970 16291 4 0- 13741 70540 3 
0-06903 58882 9 003212 73337 1 
0-01399 60398 1 0-00575 321708 


0-00224 53571 2 0-00083 61720 0 
0-00029 834760 | 0-00010 23428 0 
0-00003 38506 4 0-00001 08228 5 
0-00000 33522 0 0-00000 10077 8 
0-00000 02945 7 000000 00838 4 


0-00000 00232 7 0-00000 00063 0 
0-00000 00016 7 0-00000 00004 3 
0-00000 00001 1 0-00000 00000 3 


9 -δἝ ὧδ δυῶν βοῦν] a 


10.5 GENERATING FUNCTIONS AND DIFFERENTIAL 
EQUATIONS 


In deriving expansions using the various polynomials discussed 
above it is sometimes convenient to have them expressed in terms of 
a generating function. The most useful of these are: 


Legendre Polynomials: 
] oO 
foment =. Ba cic 
Laguerre Polynomials: 
e-zhil—-h) ὦ μη 


= Σ — L(x) ...ω.{10.5.2) 


(1 --π- ἃ) κμωρπ! 


Hermite Polynomials: 
co jn 
gizt—(h-z)") -- 5 a H,,(x) ose ( 10.5.3) 
n=0 7: 
Chebyshev Polynomials: 
detec Zeno. uss 
1—2rh-+h® ooo π(χ). ..«..{10.5.4} 
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| APPROXIMATING FUNCTIONS 
The differential equations satisfied by P,, Ly, H, and Τ᾽, are (3) 
respectively: 
(x® — 1)Pa(x) + 2xPa(x) — n(n + 1)P,(x) πο ....(10.5.5) 
xLq(x) + (1 — x)Ly(x) + n£y(x) =O ....(10.5.6) 
Hy (x) — 2xH,,(x) + 2nH,(x) =O ....(10.5.7) 
and 


(1 — x*)Th(x) — x73(x) + n®T,,(x) = 0. ....(10.5.8) 


10.6 A COMPARISON OF ACCURACY 


We conclude this brief account of the use of approximating poly- 
nomials by giving a table, due to Goopwin®), showing the relative 
number of terms required for the ordinary Taylor series and of the 
Chebyshev approximating polynomial in certain cases. 


ΕΣ : 

cos x (—2/2, 7/2) 

sin? x i-£+f 10 
ἰπ (1 +x) (0, ἘΠ) 14 


In each case the required accuracy is ten decimal places. 
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ll.l FOURIER SYNTHESIS 


One major source of numerical work in modern physics lies in 
the deduction of molecular structure from observations of the x-ray 
diffraction spectra which arise when a monochromatic beam of 
X-rays is incident, at the appropriate Bracc™ angle, on a single 
crystal. This leads to a Fourier synthesis, a type of calculation 
which occurs also in tide prediction and in astrophysics. 

For the purposes of the numerical analyst it is sufficient to note 
that Fourier synthesis calls for the evaluation of functions p(x, 9, z) 
defined by: 


l a x Ζ 
p(x, γ,. 2) = Ρ ΣΣΣΊΕΑ, k, 1)| cos [2τ(, - +k -} Π) — oy |. 
..{11.1.} 


Typical cases have 60 values of x, » and z, and thus involve the 
evaluation of (60)* values of p(x, », z). The observed coefficients 
|F(A, k, ἢ] are frequently up to 1000 in number and, in some recent 
work, have been as many as 10,000. 

Under these circumstances it is not practicable, even with a high 
speed automatic digital calculator, to evaluate the p(x, y, z) individu- 
ally, and means have been devised for the simultaneous evaluation 
of whole groups of p(x, », z); usually for fixed values of two of the 
variables and a range of values of the third. 

A very large number of different summation methods have been 
evolved, but we shall confine our attention to the method in most 
common use, which is due to Beevers and Lipson. ‘ 9) 

The Beevers-Lipson method depends upon the breaking down of 
the three dimensional synthesis (11.1.1) into a series of one dimen- 
sional syntheses of the type: 


p(x) = ἜΝ τ Αἰ Ot (am) εν λ δ 


which, as we shall presently show, can be done. 
To effect the summation (11.1.2), for a range of values of (x), 
Beevers and Lipson prepared a set of ‘strips’, one to each value of 
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A, in the integer range 1-99 and for each value of A in the range 
0-20. The sine and cosine functions are catered for separately. 
Each strip bears, upon its front surface, the information: 


(Value of A) (Sine or Cosine) (Frequency ἢ) 


(16 Values of Alta n.h) 


where n runs through the integers 0-15; the reverse side of the 
strip contains the same information, but for — A. A typical strip 
is shown in Figure 11.1.1. 


Figure 11.1.1 Beevers-Lipson strip for 32 sin 2 .3(x/a) 


3253 0 | 10| 19 | 26 | 30 82) 80 26 | 19 10 | 0 Τὸ 19 | 26 | 30 3 

The reason for the choice of interval 27/60 is a physical one which 
is related to the resolving power to be expected from the use of 
x-rays of a given wavelength. 

The use of the strips will be clear from the following simple 
example, suppose that we require: 

2 = 12 cos 27.1(x/a) + 32 sin 27.3(x/a) — 40 cos 27. μὰν τες 

..{}1.1.3} 


The appropriate strips are selected and laid together in a matrix: 


sc = δ 1 αὶ δ ὶ δ ὃ ἢ 8 ὃ 1 1Ὶ 12 18 MN SE 


1950] 12 | 12 12 [11 711} 1010] 9] 8] 1] δ᾽ δ᾽ ἐ 8] 1] Oo 
82 5 8 0 10.119 26. 80] 832. 80] 36 19 10 0 10 15) 88 ἔδ᾽ 3 
100 4 40, 37 7 ἴ3  ἐΪ 20| 88] 89] 89 8598 20) 4/13 27) 87] 40 


Ξ 88 [15] 25] 45 62 | 73 4 66 | 49 | 26 


If it is desired to extend the range beyond (2 x 27) it is merely 
necessary to reverse the cosine strips of odd frequency and the sine 
strips of even frequency, thus in the above case: 


x/a -- 80 29 28 27 26 25 24 23 22 21 20 19 18 17 16 = 


i 
12 C1 S| |i || m/w] 1 ὃ 3] 1|ὅ 1) 1| ξ| τ| 
82 5 3 Ὁ 10/19) 26/80 8880 26, 1910] 0 10 | 19 | 36 80 82 
τὖῦ Ο 4 40 | 87 | 27| 8} 4 30] 82 80) 89 3320 4 | 12 | 27 | 31 | 40 


= | 62 Δ ξ, 62 | 66 δῦ, 35 | 14 | 11 | 35 | 56 88 | 72 
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The ranges (30-45) (45-60) follow in a similar manner from the 
symmetry properties of the trigonometric functions. In more com- 
plicated summations it is more economical to perform the sine-even, 
sine-odd, cosine-even, cosine-odd summations separately and then 
to combine the results as required to extend the range. 


We now revert to the original summation (11.1.1). This may be 
written: 


, 7 x Ζ 
p(x, 7}, 2) = yr [ΕἸ cos a cos 2a( 4% + εἰ + (2) 


4 |F| sina sin 2n( hi μας 3) τ 1.1.4) 


where, for simplicity, we have written |F| for |F(h, k, ἢ} and 
a for Apg- 


In a like manner | 1.1.4 may be broken down into: 
p(x, », 2) Ἐπὶ 


PEE οος γα(: x ἘΔ 2] IF COS a@ COS Qn I + |F|sin a sin 27 /- | 
h k 


+ sin 2a( ox Ὁ 3) | |F|sin a cos Ql — |F| cosa sin 2π|- =| 
{81.1.5} 


which is of the form: 


p(x, γ. 2) = ῬΣΣμΟ, cos 2a ( + K) Ἔ μεϑὲ sin ΟΣ 4+ Ρὴ 


.. (11.1.6) 
where: 
uC: = Σ᾿ [Flos α cos 2π I ἘΝ |F| sin a sin 27 |- 1... (ILE 
ne: = S| |Flsin a COS Qn I — [ΡΠ cos a sin 2m t= |. +++ (11.1.8) 
t 


Now the summations for κε; and ,,5, can be performed directly by 
means of the Beevers-Lipson strips and the resulting sets of coeff- 
cients tabulated. The process is then repeated to give; 
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p(x, »», 2) = 


72 cos 27 (42) z[ μα cos 2π ( ἜΛΡΥΒΕ Ἦν (5) 
4. sin Qe (ι *) Σ᾿ ms: cos 27 (ἢ — κε; sin 2π (]} 


or 


| 
ρία, », 2) = γ Σχθ cos 2n( A=) + 4Sye sin 2n( n=) cv 88:8: 


where: 


Cue a Σ΄ μα cos 2m (i) + pS, sin 2n( 5) ἐν ee (11.1.10) 


Sp = | as: cos 5} = pC sin 2nr( 2). (ILL) 


b 


Thus, since μεν ,Syz can be evaluated by the Beevers-Lipson 
technique, and the final summation 11.1.9 is also in the correct form, 
the problem is solved. 


Various points of technique arise in the practical summation of 
multiple series of the type 11.1.4, notably that of ‘multiplicity 
correction’. This is necessary because the method of reduction used 
in 11.1.5-11.1.11 has the effect of including terms of the types 
(h, k, 0) (hk, 0, 0) (0, 0, 0) more than once. It is sufficient to state here 
that the corrections: 


F(h, k, 2) +1 
F(h, k, 0) +2 
F(h, 0, 0) + 4 
F(0, 0, 0) +8 


are appropriate, for further details the reader is referred to the 
specialist monographs,'4. 5) 


11.2 THE LOCATION OF MAXIMA 
One important requirement in dealing with series of the type 
envisaged in section 11.1 is that of locating accurately the maxima 
of p(x, 2), 2). This can be conveniently achieved by the methods 
of differential synthesis 7) which make unnecessary a complete 
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evaluation of p(x, y, 2) for all values of (x, y, 2). The method is essen- 
tially that of Newton-Raphson (see Chapter 9, section 9.3) applied 
to the derivatives of p(x, γ, 2). Thus, at a maximum we have: 


— See iS a ς 5.ὕ 


We denote the synthesis 11.1.1 by the shorthand notation: 
p= 5 ΣΙΕῚ cos (0 -- α) ΠῚ 
3 
whence 
2π : <i 
=— wo sin (@ —a) Ξε 


Og ἜΣ ΣΩΡΙ μα (9a) we +... 0129) 
bv; 


dp 4π ἘΣ 
<= olf sin (9 —a) = 0 


Assume that (x, y, z) is an approximation to the maximum and that 
the true value is (x + ἐν} + εν 2 + €), then, to the first order: 


— 27 s41F| sin (0 — a) + heg cos (0 —a)} =0 
aV 3 
— 5% Σ ΘΙΡῚ sin (@ —a) + keg cos (0 —a)} =O f ....(11.2.3) 
8 
— ὅπ 5 AFI sin (9 —a) + leg cos (@—a)} = 0 
cV 3 
where 
c= 2n(h HEE 15). 


From these equations we obtain: 


Année + ἄμεν + ἄμε, + An = 0 
ἄμ; + Aust; + Ante + Ay =O } ....(11-2.4) 
ἄμε, + Amey + ἄμε, + A; =0 | 
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where: 
an : 
A; = av ΣΗΙΡ s1n. (0 —a) 
Ay =— om ΣᾺ sin (9 —a) vee (11.2.5) 
Δ 
Φ ie 2 


ig dg — ὦ 
Ay = A ad ° SKF θ — 11.2.6 
ki ΙΕ — τ | | cos ( a) nea .2.6) 


ἄπ 
Arp = An = — cap TIF cos (6 —a) 


2 
Pelee ap ΣΡ cos (0 —a) 


4qr? | 

An = — jap IF COs (9 —a) PrP GE ἂν ὦ 
ἀπ 

An = — ap 2 ΕῚ cos (0 —a) 


The process as described above is second order, in practice it is 
usual to compute the coefficients A,,, μι. An, ἄμ, Au, Ay only at 
the start and at the finish of the refinement which makes the process 
formally only first order; the convergence is, however, still extremely 
rapid. 


11.3 RADIAL AND OTHER SYNTHESES 
For practical requirements it is sometimes necessary to obtain from 
the three dimensional x-ray data a radial distribution function. This 
is defined as the average of p(x, y, z) over shells of constant radius 
from the origin. The series involved is usually: 
Plu, v, w) = + DESIR, ἢ] cos 2 (ae + ee +02) 
ἀπε. ἢ V h, 1,1 ii * i b ὔ 
ὦ (tk) 
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the so-called PaTrEerson function’, or to mathematicians the ‘con- 
volution’ or ‘faltung’,® and the radial distribution function /,(r) 
is defined to be: 


4nr*p,(r)dr =| Pw v, w)dS.dr caved boca 


where dS is an element of the spherical surface, of radius r, whose 
centre is (0, 0, 0). 

Now the expression 11.3.2 may be written, by virtue of equation 
11.3.1, 


l | u v τυ 
4nr*p,(r)dr = pee ki ἢ}"] cos 2a( hf Ἑ κι +l 545. 


and the integrals can be evaluated by transforming to axes normal 
and perpendicular to the plane: 


" v w 
hee b= Us 
Thus: 


+r 


cos 2n( ).2mr.dn 


u θ᾽ 530 | 
| cos da(ae +k, 15} 45 =| 


ἘΠΕ ἘΡῚ 
: d(h, k, ἢ 


—r 


sin 2ar/d(h, k, ἢ 


— 2 es 
dee =" a, 


....(11.3.5} 
where dn is an element in the direction of the normal to 
ae ae ee 
Raat : =0 


and d(A, k, ἢ) is the length of the perpendicular from the origin on 
to the plane 
ee τυ 


It follows from 11.3.1, 11.3.2 and 11.3.3 that: 

2 Sin 2mr/d(A, k, ἢ 
Qarid(h, k,l) ἢ 
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FOURIER SYNTHESIS AND ANALYSIS 
In a similar manner it is sometimes necessary to obtain radial 
distribution functions for the two dimensional series: 
b. eeu , u v 
P(u, v) = 7 ΦΈΙΣΙΝ k, 0)|? cos γπ{(ε: +k ;) .»"«.{}1.3.5} 


and it is easily shown that the relevant function is defined by: 


2arp,(r)dr =| Pt v)dS.dr ἐφ 1.3.0 


where S is now a circle in the (u, v) plane with centre (0, 0). The 
integration is performed as before to give: 


ῥε() = A ZlFh k, 0)|*Jo2ar/d(h, k, 0) ....(11.3.7) 


where J,(z) is the zero’th order Bessel function. 


Other types of syntheses can be obtained by projecting the density 
p(x, γ, 2) contained between selected values of z, say, on to a plane 
parallel to (x, y). These, whilst of considerable value to the specialist 
crystallographer, are not of general interest and require no novelties 
of technique, the interested reader is referred to the literature. © 


11.4 FOURIER ANALYSIS 


The converse process to Fourier synthesis may be expressed, in the 
one dimensional case, by 


oo 
pls) = by + BAy cos 2n( A=) 4: Ὁ ὡς 2m = uted 
where p(x) is known in the repetition interval (0, a) and it is required 


to determine the values of the Fourier coefficients (A,, B,). 


Formally we have at once from the orthogonality of the sine and 
cosine functions: 


4, -- 2} et) cos 2ar( A=) dx (h=0,1...00) ....(11.4.2) 


By = τ ts) dia 2m ( >) Co i eer ae Ye 


To evaluate the integrals practically we make use of the Euler- 
Maclaurin integration formula (Chapter 4, section 4, equation 4.4.5), 
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When it is noted that p(0) = p(a) and that by definition the same 15 
true for all derivatives so that p\)(0) = p(a) the Euler-Maclaurin 
formula becomes in this case: 


[Ῥω con πη) dx κι 
3 | 00 + 2p(dx) cos 27 (+=) + 2p(28x) cos 2x (n.2=) +... 
+ 2p{(n — 1)8x} cos 2a in Ἢ | Ἔ ρ(0)] 


"-1 bx 
== 8x 5 p(r8x) cos 2a (tr vee. (11.44) 
ral 
since nox = a. 


Similarly: 
sii x 3 ; dx 7 
| p(x) sin )π{η5) ἀν -- Bx Σ' p(rdx) sin 2a (hr) veel LLG} 


and both of these formulae are exact. 


Whence, from 11.4.2 and 11.4.3 


= 
A, = ΤΣ p(rdx) cos 2π (ar~*) 
i ἐν. (11.5.6) 
ΝΥ πὰ bx 
B, = -: & p(rdx) sin 2x (Ὁ 
Mh γιοῦ a 


it should be noticed that by virtue of their derivation these formulae 
are significant only if kh < n/2. 

The practical computation of Fourier coefficients can be achieved 
by the use of the Beevers-Lipson strips in a manner similar to that 
used for Fourier synthesis. This follows at once if it is noticed that 
equations 11.5.6 are identical with 11.1.2 if the (x) of the latter 1s 


replaced by (A) and the (A) by (ἢ). A difficulty arises with the 


normal strips because they extend to frequencies (ἡ values) of only 
20, but this causes trouble only when Fourier coefficients of order 
greater than 10 are required. For higher values the equivalent of the 
strips are available on punched cards to frequencies of 60. 
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The process of analysis can be extended to multi-dimensional 
Fourier syntheses of the type in equation 11.1.1, the results are 
obtained in an analogous manner but are rather cumbersome and 
will not be tabulated here, (8) 
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INTEGRAL EQUATIONS 


12.1 CLASSIFICATION 


THE integral equations which have been studied most extensively by 
analysts) are those associated with the names of Fredholm and of 
Volterra, they differ only in the fact that the limits of Volterra’s 
equation contain the independent variable. 


Fredholm’s equation is usually written in the two forms: 


b 
| k(x, 9) f)dy = 2x) ....(12.1.) 


b 
| ks, 9) day = al) + fle) (02.1.2) 


and the method of solution is closely governed by the presence or 
absence of the wanted function, f(x), on the right-hand side of the 
equation. The function k(x, y) is known as the ‘kernel’ and, just 
as is the case in the classical analytical theory, the technique of 
solution is simplified when k(x, y) issymmetricin x and y. Associated 
with the Fredholm equations is the eigenvalue problem: 


b 
al K(x, ») (9) dy = 1.) -+++(12.1.3) 


where, not only is f(x) unknown, but a solution exists only for 
discrete values of A which have also to be determined. 


The Volterra equations are 
z 
| Gs fday = a) (12.14) 
be 


] ἀφο) Λοὴῷ = els) ἘΜΟῚ τς ATES) 


and it is intuitively evident that their solution will resemble that of 
one-point boundary differential equations, while that of the Fred- 
holm equations will be more akin to the two-point type. 
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An integral equation is said to be non-singular when both αὶ and 
J, as well as the limits of integration, are finite and continuous. 

Other integral equations derive from Fourier’s theorem and from 
the Laplace transform ; since they are more properly a part of 
analysis they will not be considered further here. 

Two classical equations are those of Abel and of Schlémilch. 
Abel’s equation is: 


= : 
"ὁ 1.) ῦ«-μ-ὶ] | 
which can be shown) to have the solution: 
__ sin (ur) ἃ Y  g(x)dx | 
cS Ε-: Ὡἰἱ ΓΞ xe ἐν. {13.1.5} 


so that, when g(x) is known, f(y) can be found either analytically or 
by numerical integration. 
Schlémilch’s equation is: 


τ 
g(x) = =| f(x sin θ)θ --πεξ χ ξβπ ....(12.1.8) 
0 
and the solution is“: , 
πὶ 
fiz) = g(0) + 4] σ'(α sin 8)d0 ....(12.1.9) 
0 


which is also amenable to treatment. 

Equations occur in which both integrals and derivatives are 
present, and these are usually called integro-differential equations, 
The method of numerical solution depends upon the type of bound- 
ary conditions which have to be satisfied, but otherwise follows 
closely upon the lines which will be indicated in sections 12.2 and 
12.3. 


12.2 vOLTERRA’S EQUATION 


In the first place we may note that an equation of the type 12.1.4 
can always be reduced to one of type 12.1.5 by a single differentia- 
tion with respect to x if k(x, x) 4 0, or by a series of such differentia- 
tions until a derivative is found such that K(x, x) τέ 0. 

The solution now proceeds in two parts, first initial values are 
obtained by means of a ‘Taylor series expansion : 


f(a +x) =fla) + χΓ(α) ἰ ἐχ (a) +. - - 
198 


VOLTERRA’S EQUATION 
where, from equation 12.1.5, we have: 
f(a) = — g(a) 
f'(@) = — κ΄ (α) + Ka, a) f(a) 
Γ΄ (α) = — g"(a) + Ka, a) f(a) + Κ΄ (α, a) f(a) + Ka, a) Γ΄ (α) 


Ἀ-ἸΓ qn-i-r 
Fora) = — g(a) +°E | Fae UH, 2) Δ} |, 
γα}... =a. 
vine (BBB) 
Next we take Gregory’s formula for the integral which may be 
derived from the Euler-Maclaurin formula 4.4.5 by substituting 
for the derivatives in terms of forward and backward differences. 
The Gregory formula is: 
a+noz 
| fe)ax = 
a 
Sx[4 fla) + fla + 8x) +... +f{a + (n—1)88} + fla + n8x)] 
+ δα[γ;δ — fd? + ἡκάδ" -- τὸτδ' ΘΟ. . IS 
-- δας + AV? + ἡδον" Ὁ τῆσδ ἘΠ... .1.1α + 26x) 
a sso (heeded) 
and it is seen that, by using it to a prescribed number of terms, the 
given integral can be expressed in the form: 


pa+ πδα a 
| fls)dx = Σ 8xA, fla + 18) (12.2.3) 


where the coefficients A, depend only upon the number of differ- 
ences to which 12.2.2 is taken. 
By means of 12.2.3 the integral equation 12.1.5 may be written : 
Sx Σ A,k(a + ndx, a +18x) fla -+ r8x) = g(a + nBx) +f(a-+ndx) 
r=O 
or: 
[1 — 8xA,k(a +- ndx, a + ndx)] f(a + ndx) = — g(a + ndx) 
4 8x F Aba + abe, a 4184) fla-+ 18x) ....(12.2.4) 
r= 


whence, if the first (n) values of f(a + 78x) are known, a further 
value may be obtained. The process can now be repeated, using the 
values f(a + ndx) . . . f(a + 5x), and the solution thus continued. 
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Care must be taken that differences of sufficiently high order be 
included in equation 12.2.2 for the rate of growth of error to be 
adequately small in the range over which a solution is required. For 
an indication of how this is done, and also for suggestions as to 
methods of estimating the error in particular cases, the reader is 
referred to the excellent paper of Fox and Goopwin™), 


12.3 FREDHOLM’S EQUATION 


The solution of the Fredholm equation of the first kind, 12.1.1, is 
complicated by the fact that solutions may not be possible when k 
and g are connected in certain ways. An obvious example of this 


arises when k(x, y) has the form Σ X,(x).Y,(y), in which case 
r=1 


equation 12.1.1 becomes: 


2 GX«As) = a(*) τον PS) 


where: 


b 
G, -| γ,Ο) f(a) ey. - +++ (12.3.2) 


Here no solution is possible unless g(x) has the form 12.3.1 but, 
in this case, any solution is possible for which equation 12.3.2 is 
satisfied. Fox and Goopwin“) conclude that no really satisfactory 
numerical method exists for the accurate solution of equations of 
this kind. 

For Fredholm’s equation of the second kind, 12.1.2, the position 
is more satisfactory. The approach is again to represent the integral 
by a finite difference approximation of the type given in equations 
12.2.2 and 12.2.3, but from here there exists a choice of two 
methods. 

In the first a sufficient number of differences are retained in 
equation 12.2.2 to ensure that the reduced form 12.2.3 gives an 
adequate representation of the integral over the range considered, 
in the second method equation 12.2.2 is written in the form: 


a+ nbz 
| fla)dx = Be(8 fla) Ἔα" δὴ +... 


+f{a+ (n—1)8x} + 4 flat πδα)] ἘΞ ....(12.3.3) 
where & is a correction term compounded of the differences given 
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in equation 12.2.2. The integral equation is now written in one of 
the forms: 


By Σ 4,105, α + 89) fla + τὸ») = g(x) +fla) ....(.3.3.4) 
or: 
Sy[BK(x, a) fla) + k(x, a +8) χα - δ) +... 
+ k{x, a + (ἡ — 1)8y} f{a + (n — 1)8y} 


+ k(x, a + ndy) f(a + ndy)] + 8, = g(x) +f (+). 
... « (12.3.5) 


We notice that a + ndy = ὃ and then proceed to solve the (n + 1) 
simultaneous linear equations, which result from taking x =a, a +- 8x, 

. a -+ ndx in 12.3.4, for the (n +- 1) values of f(x). ‘This method has 
the disadvantage that i it is necessary to keep sufficient differences in 
deriving equation 12.2.3 to ensure the required final accuracy and 
since the behaviour of the solution is in general unknown, this may 
lead to wasted effort. 

Fox and Goopwin(® have suggested a modification of their 
method for the solution of ordinary differential equations (see page 
68). In the present instance they solve the equations which result 
from substituting x = a, a + dx, . . . a + ndx in 12.3.5, initially neg- 
lecting the corrections &,. The set of approximations f(a), f(a +- 8x) 

εν f(a +- ndx) are then used to estimate the values of &, and 
the solution is re-computed to give a better approximation f(a), 
f(a + 8x), etc. This process is repeated until no significant change 
is produced between two successive approximations, 

Alternative methods which have been suggested for the solution 
of equations of the Fredholm type include the deferred approach to 
the limit technique of RicHarpson™), and the analogue of Picard’s 
method for ordinary differential equations (see page 57) which sets 
up an iterative sequence: 


b 
Fs) = — gle) ΕἾ συλ τ ο)ά., ....(12.8.6) 


Since it is necessary to calculate the value of each iterate at a suffici- 
ent number of points to enable the next integrations to be carried 
out to an adequate accuracy the method is likely to be tedious. 


12.4 THE EIGENVALUE PROBLEM 


The solution of the eigenvalue problem typified by equation 12.1.3 
follows closely upon the lines just suggested for solving the Fredholm 
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equation 12.1.2. The equation is put into one of the finite difference 
forms : 


2 Ads, a + τὸν) fla + By) = Δὲν ....(12.4.1) 
Or : 
k(x, a) f(a) + k(x, a + by) f(a + dy) +... 
+ A{x, a + (n — 1)dy} f{a + (n — 1)dy} 
+ $k(x, a + ndy) fla + ndy) = f(x) /ABy — B/Sy 


awe haben) 
and the sets of stmultaneous equations which result from taking 
x=a,a-+6x,a+23x . . . 4- πδχ are then used as a basis of 
computation. 


For the equations which derive from 12.4.1 the values of A and 
their associated vectors are determined in the manner described in 
Chapter 7 section 7.6 since it is evident that non-zero solutions exist 
only if the determinant: 


Aok(0, 0) — 1/Aby, A,K(O,1),. . . . « An&(0, 2) 
Ayk(1, 0) » A, A(T, 1) τὸν, . . Ayk(1, 2) 
Agk(n, 0) » Ajk(n,1),. . .  . Ank(n,n) — 1/A8y 


is zero. We have used the shorthand notation: 
k(r, s) = k(a + rox, a + sdy). 


This technique pre-supposes a knowledge of the behaviour of the 
solution to enable appropriate values of the coefficients A, to be 
chosen. The second method first obtains rough values of A under the 
assumption that the corrections &, are zero and then uses each of 
these values of A to derive a trial vector f(a), f(a + 8x) . . . f(a+-n8x). 
From this trial vector the corrections &, can be computed and then, 
in turn, an improved value of A derived; the latter operation is best 
carried out by means of the second order process given in Chapter 7 
section 6 equation 7.6.2. | 

The whole process is a cyclic one which is terminated as soon as 
the change between two steps is less than the desired value. 
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12.5 MONTE CARLO METHODS 


Since the methods of solution for integral equations which we have 
just outlined aim at reducing the equation to a set of linear simul- 
taneous equations, it is natural to suppose that, just as is the case for 
the latter, there will be Monte Carlo methods for solving integral 
equations. 


The Fredholm equation arises in the study of the behaviour of 
neutrons which are incident on a plate of material in which collisions 
can give rise either to more or to less than one particle. This sug- 
gests (8) the following Monte Carlo process for solving an equation 
of the type given in 12.1.2; first the equation is normalized so 
that : 


b ου 
| g(x)dx=1 and | E(<,sar <1... (TRS) 
a --(οὐ 


f(x) now represents the density of collisions in a population of 
particles in (a, 5) when it is assumed that the density of first collisions 
is g(x). In normalizing k(x, y) a numerical factor, A, will have been 
introduced and the equation can be written: 


b 
fle) = en(x) + a| kin(x, 9) flg)dy ....(12.5.2) 


where the subscript (n) indicates that the function has been 
normalized in accord with 12.5.1, and the number A is interpreted 
to be the mean number of particles which remain after each 
collision. 


To solve the problem, we first pick a random number ry from a 
population which is uniformly distributed in (0, 1), a first collision 
position, x,, is then determined from the equation: 


[ g(a)de “ες ....(12.5.3) 


and a record is made that one collision has occurred at x,. Now A 
will not, in general, be a whole number, so that to determine the 
number of particles which result from the collision, a statistical pro- 
cedure must be used. A simple method is to take the number of 
particles to be that integer just less than or just greater than A 
according to whether a second random number, 1, is less than, or 
greater than, the fractional part of A. 
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Each particle which results from the first collision is now followed, 
and second collision positions, x., are determined from: 


Zs 
| K(x, y)dy = τς νι... -(12.5.4) 


-οὐ 


where r, is again a random number in (0, 1). 


cb 
When τς >| k(x,, y)dy the particle is assumed to have escaped from 
a 

(a, δ) and is no longer of interest ; when all of the particles which 
resulted from the first collision have been followed to their termina- 
tion (either by attenuation or by escape) a new particle is started and 
the process repeated. After a number of repetitions of this process a 
graphical record will be available on which is marked each collision 
which occurred in (a, δ); a smooth curve drawn through this gives the 
required approximation to f(x). 

In a practical application of the above process a sample of 100 
starting neutrons yielded a final accuracy of about 10 per cent. It 
is perhaps worth mentioning that equations 12.5.3 and 12.5.4 are 
normally solved by means of previously constructed graphs. 
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